Feb 3 14:51:33 oss2 kernel: Linux version 2.6.18-238.12.1.el5_lustre.gce5e033 (jenkins@client-3.lab.whamcloud.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Thu Jun 23 20:11:14 PDT 2011 Feb 3 14:51:33 oss2 kernel: Command line: ro root=LABEL=/ crashkernel=128M@16M console=ttyS1,115200 console=tty0 Feb 3 14:51:33 oss2 kernel: BIOS-provided physical RAM map: Feb 3 14:51:33 oss2 kernel: BIOS-e820: 0000000000010000 - 000000000009bc00 (usable) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 000000000009bc00 - 00000000000a0000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 0000000000100000 - 00000000bf760000 (usable) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9 Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) Feb 3 14:51:33 oss2 kernel: BIOS-e820: 0000000100000000 - 0000000c40000000 (usable) Feb 3 14:51:33 oss2 kernel: DMI present. Feb 3 14:51:33 oss2 kernel: ACPI: RSDP (v002 ACPIAM ) @ 0x00000000000fa0c0 Feb 3 14:51:33 oss2 kernel: ACPI: XSDT (v001 SMCI 0x20110407 MSFT 0x00000097) @ 0x00000000bf770100 Feb 3 14:51:33 oss2 kernel: ACPI: FADT (v004 040711 FACP1934 0x20110407 MSFT 0x00000097) @ 0x00000000bf770290 Feb 3 14:51:33 oss2 kernel: ACPI: MADT (v002 040711 APIC1934 0x20110407 MSFT 0x00000097) @ 0x00000000bf770390 Feb 3 14:51:33 oss2 kernel: ACPI: MCFG (v001 040711 OEMMCFG 0x20110407 MSFT 0x00000097) @ 0x00000000bf7704d0 Feb 3 14:51:33 oss2 kernel: ACPI: SLIT (v001 040711 OEMSLIT 0x20110407 MSFT 0x00000097) @ 0x00000000bf770510 Feb 3 14:51:33 oss2 kernel: ACPI: OEMB (v001 040711 OEMB1934 0x20110407 MSFT 0x00000097) @ 0x00000000bf77e040 Feb 3 14:51:33 oss2 kernel: ACPI: SRAT (v002 040711 OEMSRAT 0x00000001 INTL 0x00000001) @ 0x00000000bf77a6c0 Feb 3 14:51:33 oss2 kernel: ACPI: HPET (v001 040711 OEMHPET 0x20110407 MSFT 0x00000097) @ 0x00000000bf77a910 Feb 3 14:51:33 oss2 kernel: ACPI: DMAR (v001 AMI OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x00000000bf77e0d0 Feb 3 14:51:33 oss2 kernel: ACPI: SSDT (v001 DpgPmm CpuPm 0x00000012 INTL 0x20051117) @ 0x00000000bf785510 Feb 3 14:51:33 oss2 kernel: ACPI: EINJ (v001 AMIER AMI_EINJ 0x20110407 MSFT 0x00000097) @ 0x00000000bf77a950 Feb 3 14:51:33 oss2 kernel: ACPI: BERT (v001 AMIER AMI_BERT 0x20110407 MSFT 0x00000097) @ 0x00000000bf77aae0 Feb 3 14:51:33 oss2 kernel: ACPI: ERST (v001 AMIER AMI_ERST 0x20110407 MSFT 0x00000097) @ 0x00000000bf77ab10 Feb 3 14:51:33 oss2 kernel: ACPI: HEST (v001 AMIER ABC_HEST 0x20110407 MSFT 0x00000097) @ 0x00000000bf77acc0 Feb 3 14:51:33 oss2 kernel: ACPI: DSDT (v002 SF280 SF280000 0x00000000 INTL 0x20051117) @ 0x0000000000000000 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 16 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 18 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 20 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 17 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 19 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 0 -> APIC 21 -> Node 0 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 32 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 34 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 36 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 48 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 50 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 52 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 33 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 35 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 37 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 49 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 51 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: PXM 1 -> APIC 53 -> Node 1 Feb 3 14:51:33 oss2 kernel: SRAT: Node 0 PXM 0 0-a0000 Feb 3 14:51:33 oss2 kernel: SRAT: Node 0 PXM 0 0-c0000000 Feb 3 14:51:33 oss2 kernel: SRAT: Node 0 PXM 0 0-640000000 Feb 3 14:51:33 oss2 kernel: SRAT: Node 1 PXM 1 640000000-c40000000 Feb 3 14:51:33 oss2 kernel: NUMA: Using 30 for the hash shift. Feb 3 14:51:33 oss2 kernel: Bootmem setup node 0 0000000000000000-0000000640000000 Feb 3 14:51:33 oss2 kernel: bootmap start 100000 pages 200 Feb 3 14:51:33 oss2 kernel: Bootmem setup node 1 0000000640000000-0000000c40000000 Feb 3 14:51:33 oss2 kernel: bootmap start 640006000 pages 192 Feb 3 14:51:33 oss2 kernel: Setting up node 0 0-640000 Feb 3 14:51:33 oss2 kernel: On node 0 totalpages: 6197994 Feb 3 14:51:33 oss2 kernel: DMA zone: 2386 pages, LIFO batch:0 Feb 3 14:51:33 oss2 kernel: DMA32 zone: 765848 pages, LIFO batch:31 Feb 3 14:51:33 oss2 kernel: Normal zone: 5429760 pages, LIFO batch:31 Feb 3 14:51:33 oss2 kernel: Setting up node 1 640000-c40000 Feb 3 14:51:33 oss2 kernel: On node 1 totalpages: 6205440 Feb 3 14:51:33 oss2 kernel: Normal zone: 6205440 pages, LIFO batch:31 Feb 3 14:51:33 oss2 kernel: ACPI: PM-Timer IO Port: 0x808 Feb 3 14:51:33 oss2 kernel: ACPI: Local APIC address 0xfee00000 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Feb 3 14:51:33 oss2 kernel: Processor #0 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) Feb 3 14:51:33 oss2 kernel: Processor #2 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) Feb 3 14:51:33 oss2 kernel: Processor #4 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled) Feb 3 14:51:33 oss2 kernel: Processor #16 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled) Feb 3 14:51:33 oss2 kernel: Processor #18 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled) Feb 3 14:51:33 oss2 kernel: Processor #20 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x20] enabled) Feb 3 14:51:33 oss2 kernel: Processor #32 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x22] enabled) Feb 3 14:51:33 oss2 kernel: Processor #34 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x24] enabled) Feb 3 14:51:33 oss2 kernel: Processor #36 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x30] enabled) Feb 3 14:51:33 oss2 kernel: Processor #48 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) Feb 3 14:51:33 oss2 kernel: Processor #50 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x34] enabled) Feb 3 14:51:33 oss2 kernel: Processor #52 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x01] enabled) Feb 3 14:51:33 oss2 kernel: Processor #1 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x03] enabled) Feb 3 14:51:33 oss2 kernel: Processor #3 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x05] enabled) Feb 3 14:51:33 oss2 kernel: Processor #5 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x11] enabled) Feb 3 14:51:33 oss2 kernel: Processor #17 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x13] enabled) Feb 3 14:51:33 oss2 kernel: Processor #19 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x15] enabled) Feb 3 14:51:33 oss2 kernel: Processor #21 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x21] enabled) Feb 3 14:51:33 oss2 kernel: Processor #33 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x23] enabled) Feb 3 14:51:33 oss2 kernel: Processor #35 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] enabled) Feb 3 14:51:33 oss2 kernel: Processor #37 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x31] enabled) Feb 3 14:51:33 oss2 kernel: Processor #49 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x33] enabled) Feb 3 14:51:33 oss2 kernel: Processor #51 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x35] enabled) Feb 3 14:51:33 oss2 kernel: Processor #53 6:12 APIC version 21 Feb 3 14:51:33 oss2 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) Feb 3 14:51:33 oss2 kernel: ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0]) Feb 3 14:51:33 oss2 kernel: IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23 Feb 3 14:51:33 oss2 kernel: ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24]) Feb 3 14:51:33 oss2 kernel: IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47 Feb 3 14:51:33 oss2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec80000] gsi_base[48]) Feb 3 14:51:33 oss2 kernel: IOAPIC[2]: apic_id 8, version 32, address 0xfec80000, GSI 48-71 Feb 3 14:51:33 oss2 kernel: ACPI: IOAPIC (id[0x09] address[0xfec80400] gsi_base[72]) Feb 3 14:51:33 oss2 kernel: IOAPIC[3]: apic_id 9, version 32, address 0xfec80400, GSI 72-95 Feb 3 14:51:33 oss2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Feb 3 14:51:33 oss2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 20 low level) Feb 3 14:51:33 oss2 kernel: ACPI: IRQ0 used by override. Feb 3 14:51:33 oss2 kernel: ACPI: IRQ2 used by override. Feb 3 14:51:33 oss2 kernel: Setting APIC routing to physical flat Feb 3 14:51:33 oss2 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000 Feb 3 14:51:33 oss2 kernel: Using ACPI (MADT) for SMP configuration information Feb 3 14:51:33 oss2 kernel: Nosave address range: 000000000009b000 - 000000000009c000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 000000000009c000 - 00000000000a0000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000000a0000 - 00000000000e6000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000000e6000 - 0000000000100000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf760000 - 00000000bf76e000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf76e000 - 00000000bf770000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf770000 - 00000000bf77e000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf77e000 - 00000000bf7d0000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf7d0000 - 00000000bf7e0000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf7e0000 - 00000000bf7ec000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000bf7ec000 - 00000000c0000000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000c0000000 - 00000000e0000000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000f0000000 - 00000000fee00000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000fee01000 - 00000000ffc00000 Feb 3 14:51:33 oss2 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000 Feb 3 14:51:33 oss2 kernel: Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) Feb 3 14:51:33 oss2 kernel: SMP: Allowing 24 CPUs, 0 hotplug CPUs Feb 3 14:51:33 oss2 kernel: Built 2 zonelists. Total pages: 12403434 Feb 3 14:51:33 oss2 kernel: Kernel command line: ro root=LABEL=/ crashkernel=128M@16M console=ttyS1,115200 console=tty0 Feb 3 14:51:33 oss2 kernel: Initializing CPU#0 Feb 3 14:51:33 oss2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Feb 3 14:51:33 oss2 kernel: Console: colour VGA+ 80x25 Feb 3 14:51:33 oss2 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Feb 3 14:51:33 oss2 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Feb 3 14:51:33 oss2 kernel: Checking aperture... Feb 3 14:51:33 oss2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Feb 3 14:51:33 oss2 kernel: Placing software IO TLB between 0xf07f000 - 0x1307f000 Feb 3 14:51:33 oss2 kernel: Memory: 49313660k/51380224k available (2594k kernel code, 1008688k reserved, 1652k data, 224k init) Feb 3 14:51:33 oss2 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 6133.76 BogoMIPS (lpj=3066881) Feb 3 14:51:33 oss2 kernel: Security Framework v1.0.0 initialized Feb 3 14:51:33 oss2 kernel: SELinux: Initializing. Feb 3 14:51:33 oss2 kernel: SELinux: Starting in permissive mode Feb 3 14:51:33 oss2 kernel: selinux_register_security: Registering secondary module capability Feb 3 14:51:33 oss2 kernel: Capability LSM initialized as secondary Feb 3 14:51:33 oss2 kernel: Mount-cache hash table entries: 256 Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 0/0 -> Node 0 Feb 3 14:51:33 oss2 kernel: using mwait in idle threads. Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 0 Feb 3 14:51:33 oss2 kernel: CPU0: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to UP code Feb 3 14:51:33 oss2 kernel: ACPI: Core revision 20060707 Feb 3 14:51:33 oss2 kernel: Using local APIC timer interrupts. Feb 3 14:51:33 oss2 kernel: Detected 8.333 MHz APIC timer. Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 1/24 APIC 0x2 Feb 3 14:51:33 oss2 kernel: Initializing CPU#1 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.50 BogoMIPS (lpj=3066753) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 1/2 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 1 Feb 3 14:51:33 oss2 kernel: CPU1: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 2/24 APIC 0x4 Feb 3 14:51:33 oss2 kernel: Initializing CPU#2 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.46 BogoMIPS (lpj=3066733) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 2/4 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 2 Feb 3 14:51:33 oss2 kernel: CPU2: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 3/24 APIC 0x10 Feb 3 14:51:33 oss2 kernel: Initializing CPU#3 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.62 BogoMIPS (lpj=3066814) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 3/10 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 8 Feb 3 14:51:33 oss2 kernel: CPU3: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 4/24 APIC 0x12 Feb 3 14:51:33 oss2 kernel: Initializing CPU#4 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.73 BogoMIPS (lpj=3066869) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 4/12 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 9 Feb 3 14:51:33 oss2 kernel: CPU4: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 5/24 APIC 0x14 Feb 3 14:51:33 oss2 kernel: Initializing CPU#5 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.59 BogoMIPS (lpj=3066798) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 5/14 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 10 Feb 3 14:51:33 oss2 kernel: CPU5: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 6/24 APIC 0x20 Feb 3 14:51:33 oss2 kernel: Initializing CPU#6 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.45 BogoMIPS (lpj=3066728) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 6/20 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 0 Feb 3 14:51:33 oss2 kernel: CPU6: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 7/24 APIC 0x22 Feb 3 14:51:33 oss2 kernel: Initializing CPU#7 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.44 BogoMIPS (lpj=3066724) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 7/22 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 1 Feb 3 14:51:33 oss2 kernel: CPU7: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 8/24 APIC 0x24 Feb 3 14:51:33 oss2 kernel: Initializing CPU#8 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.46 BogoMIPS (lpj=3066731) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 8/24 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 2 Feb 3 14:51:33 oss2 kernel: CPU8: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 9/24 APIC 0x30 Feb 3 14:51:33 oss2 kernel: Initializing CPU#9 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.47 BogoMIPS (lpj=3066737) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 9/30 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 8 Feb 3 14:51:33 oss2 kernel: CPU9: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 10/24 APIC 0x32 Feb 3 14:51:33 oss2 kernel: Initializing CPU#10 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.44 BogoMIPS (lpj=3066724) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 10/32 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 9 Feb 3 14:51:33 oss2 kernel: CPU10: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 11/24 APIC 0x34 Feb 3 14:51:33 oss2 kernel: Initializing CPU#11 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.46 BogoMIPS (lpj=3066731) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 11/34 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 10 Feb 3 14:51:33 oss2 kernel: CPU11: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 12/24 APIC 0x1 Feb 3 14:51:33 oss2 kernel: Initializing CPU#12 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.44 BogoMIPS (lpj=3066723) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 12/1 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 0 Feb 3 14:51:33 oss2 kernel: CPU12: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 13/24 APIC 0x3 Feb 3 14:51:33 oss2 kernel: Initializing CPU#13 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.45 BogoMIPS (lpj=3066728) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 13/3 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 1 Feb 3 14:51:33 oss2 kernel: CPU13: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 14/24 APIC 0x5 Feb 3 14:51:33 oss2 kernel: Initializing CPU#14 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.46 BogoMIPS (lpj=3066730) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 14/5 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 2 Feb 3 14:51:33 oss2 kernel: CPU14: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 15/24 APIC 0x11 Feb 3 14:51:33 oss2 kernel: Initializing CPU#15 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.54 BogoMIPS (lpj=3066774) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 15/11 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 8 Feb 3 14:51:33 oss2 kernel: CPU15: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 16/24 APIC 0x13 Feb 3 14:51:33 oss2 kernel: Initializing CPU#16 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.64 BogoMIPS (lpj=3066821) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 16/13 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 9 Feb 3 14:51:33 oss2 kernel: CPU16: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 17/24 APIC 0x15 Feb 3 14:51:33 oss2 kernel: Initializing CPU#17 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.81 BogoMIPS (lpj=3066907) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 17/15 -> Node 0 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 0 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 10 Feb 3 14:51:33 oss2 kernel: CPU17: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 18/24 APIC 0x21 Feb 3 14:51:33 oss2 kernel: Initializing CPU#18 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.90 BogoMIPS (lpj=3066953) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 18/21 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 0 Feb 3 14:51:33 oss2 kernel: CPU18: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 19/24 APIC 0x23 Feb 3 14:51:33 oss2 kernel: Initializing CPU#19 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.72 BogoMIPS (lpj=3066864) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 19/23 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 1 Feb 3 14:51:33 oss2 kernel: CPU19: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 20/24 APIC 0x25 Feb 3 14:51:33 oss2 kernel: Initializing CPU#20 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.65 BogoMIPS (lpj=3066826) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 20/25 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 2 Feb 3 14:51:33 oss2 kernel: CPU20: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 21/24 APIC 0x31 Feb 3 14:51:33 oss2 kernel: Initializing CPU#21 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.47 BogoMIPS (lpj=3066739) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 21/31 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 8 Feb 3 14:51:33 oss2 kernel: CPU21: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 22/24 APIC 0x33 Feb 3 14:51:33 oss2 kernel: Initializing CPU#22 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.47 BogoMIPS (lpj=3066739) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 22/33 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 9 Feb 3 14:51:33 oss2 kernel: CPU22: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: SMP alternatives: switching to SMP code Feb 3 14:51:33 oss2 kernel: Booting processor 23/24 APIC 0x35 Feb 3 14:51:33 oss2 kernel: Initializing CPU#23 Feb 3 14:51:33 oss2 kernel: Calibrating delay using timer specific routine.. 6133.47 BogoMIPS (lpj=3066736) Feb 3 14:51:33 oss2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Feb 3 14:51:33 oss2 kernel: CPU: L2 cache: 256K Feb 3 14:51:33 oss2 kernel: CPU: L3 cache: 12288K Feb 3 14:51:33 oss2 kernel: CPU 23/35 -> Node 1 Feb 3 14:51:33 oss2 kernel: CPU: Physical Processor ID: 1 Feb 3 14:51:33 oss2 kernel: CPU: Processor Core ID: 10 Feb 3 14:51:33 oss2 kernel: CPU23: Thermal monitoring enabled (TM1) Feb 3 14:51:33 oss2 kernel: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz stepping 02 Feb 3 14:51:33 oss2 kernel: Brought up 24 CPUs Feb 3 14:51:33 oss2 kernel: CPU#0: NMI watchdog performance counter calibration - 2675->2694 Feb 3 14:51:33 oss2 kernel: CPU#1: NMI watchdog performance counter calibration - 66->86 Feb 3 14:51:33 oss2 kernel: CPU#2: NMI watchdog performance counter calibration - 65->85 Feb 3 14:51:33 oss2 kernel: CPU#3: NMI watchdog performance counter calibration - 65->85 Feb 3 14:51:33 oss2 kernel: CPU#4: NMI watchdog performance counter calibration - 64->84 Feb 3 14:51:33 oss2 kernel: CPU#5: NMI watchdog performance counter calibration - 65->85 Feb 3 14:51:33 oss2 kernel: CPU#6: NMI watchdog performance counter calibration - 64->84 Feb 3 14:51:33 oss2 kernel: CPU#7: NMI watchdog performance counter calibration - 64->84 Feb 3 14:51:33 oss2 kernel: CPU#8: NMI watchdog performance counter calibration - 63->83 Feb 3 14:51:33 oss2 kernel: CPU#9: NMI watchdog performance counter calibration - 63->83 Feb 3 14:51:33 oss2 kernel: CPU#10: NMI watchdog performance counter calibration - 63->83 Feb 3 14:51:33 oss2 kernel: CPU#11: NMI watchdog performance counter calibration - 63->83 Feb 3 14:51:33 oss2 kernel: CPU#12: NMI watchdog performance counter calibration - 64->84 Feb 3 14:51:33 oss2 kernel: CPU#13: NMI watchdog performance counter calibration - 62->82 Feb 3 14:51:33 oss2 kernel: CPU#14: NMI watchdog performance counter calibration - 62->82 Feb 3 14:51:33 oss2 kernel: CPU#15: NMI watchdog performance counter calibration - 62->82 Feb 3 14:51:33 oss2 kernel: CPU#16: NMI watchdog performance counter calibration - 62->82 Feb 3 14:51:33 oss2 kernel: CPU#17: NMI watchdog performance counter calibration - 62->82 Feb 3 14:51:33 oss2 kernel: CPU#18: NMI watchdog performance counter calibration - 61->81 Feb 3 14:51:33 oss2 kernel: CPU#19: NMI watchdog performance counter calibration - 61->81 Feb 3 14:51:33 oss2 kernel: CPU#20: NMI watchdog performance counter calibration - 61->81 Feb 3 14:51:33 oss2 kernel: CPU#21: NMI watchdog performance counter calibration - 60->80 Feb 3 14:51:33 oss2 kernel: CPU#22: NMI watchdog performance counter calibration - 60->80 Feb 3 14:51:33 oss2 kernel: CPU#23: NMI watchdog performance counter calibration - 60->80 Feb 3 14:51:33 oss2 kernel: NMI watchdog testing PASSED. Feb 3 14:51:33 oss2 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer. Feb 3 14:51:33 oss2 kernel: time.c: Detected 3066.881 MHz processor. Feb 3 14:51:33 oss2 kernel: sizeof(vma)=176 bytes Feb 3 14:51:33 oss2 kernel: sizeof(page)=56 bytes Feb 3 14:51:33 oss2 kernel: sizeof(inode)=576 bytes Feb 3 14:51:33 oss2 kernel: sizeof(dentry)=216 bytes Feb 3 14:51:33 oss2 kernel: sizeof(ext3inode)=776 bytes Feb 3 14:51:33 oss2 kernel: sizeof(buffer_head)=96 bytes Feb 3 14:51:33 oss2 kernel: sizeof(skbuff)=248 bytes Feb 3 14:51:33 oss2 kernel: migration_cost=1,20,1010 Feb 3 14:51:33 oss2 kernel: checking if image is initramfs... it is Feb 3 14:51:33 oss2 kernel: Freeing initrd memory: 3412k freed Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 16 Feb 3 14:51:33 oss2 kernel: ACPI: bus type pci registered Feb 3 14:51:33 oss2 kernel: PCI: Using MMCONFIG at e0000000 Feb 3 14:51:33 oss2 kernel: ACPI: Interpreter enabled Feb 3 14:51:33 oss2 kernel: ACPI: Using IOAPIC for interrupt routing Feb 3 14:51:33 oss2 kernel: ACPI: No dock devices found. Feb 3 14:51:33 oss2 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Feb 3 14:51:33 oss2 kernel: PCI: PXH quirk detected, disabling MSI for SHPC device Feb 3 14:51:33 oss2 kernel: PCI: PXH quirk detected, disabling MSI for SHPC device Feb 3 14:51:33 oss2 kernel: PCI: Transparent bridge - 0000:00:1e.0 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE3._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE5._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE7._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9.PXHA._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE9.PXHB._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE1._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P8._PRT] Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 *7 10 11 12 14 15) Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *10 11 12 14 15) Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 *11 12 14 15) Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 7 10 11 12 14 15) Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15) Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 14 *15) Feb 3 14:51:33 oss2 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Feb 3 14:51:33 oss2 kernel: pnp: PnP ACPI init Feb 3 14:51:33 oss2 kernel: pnp: PnP ACPI: found 15 devices Feb 3 14:51:33 oss2 kernel: usbcore: registered new driver usbfs Feb 3 14:51:33 oss2 kernel: usbcore: registered new driver hub Feb 3 14:51:33 oss2 kernel: PCI: Using ACPI for IRQ routing Feb 3 14:51:33 oss2 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report Feb 3 14:51:33 oss2 kernel: NetLabel: Initializing Feb 3 14:51:33 oss2 kernel: NetLabel: domain hash size = 128 Feb 3 14:51:33 oss2 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Feb 3 14:51:33 oss2 kernel: NetLabel: unlabeled traffic allowed by default Feb 3 14:51:33 oss2 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0, 0 Feb 3 14:51:33 oss2 kernel: hpet0: 4 64-bit timers, 14318180 Hz Feb 3 14:51:33 oss2 kernel: DMAR:Host address width 40 Feb 3 14:51:33 oss2 kernel: DMAR:DRHD base: 0x000000fbffe000 flags: 0x1 Feb 3 14:51:33 oss2 kernel: IOMMU fbffe000: ver 1:0 cap c90780106f0462 ecap f020f6 Feb 3 14:51:33 oss2 kernel: DMAR:RMRR base: 0x000000000e6000 end: 0x000000000e9fff Feb 3 14:51:33 oss2 kernel: DMAR:RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff Feb 3 14:51:33 oss2 kernel: DMAR:ATSR flags: 0x0 Feb 3 14:51:33 oss2 kernel: PCI-GART: No AMD northbridge found. Feb 3 14:51:33 oss2 kernel: pnp: 00:01: iomem range 0xfbf00000-0xfbffffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:01: iomem range 0xfc000000-0xfcffffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:01: iomem range 0xfd000000-0xfdffffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:01: iomem range 0xfe000000-0xfebfffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:08: iomem range 0xfed1c000-0xfed1ffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:08: iomem range 0xfed20000-0xfed3ffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:08: iomem range 0xfed40000-0xfed8ffff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0a: iomem range 0xfec00000-0xfec00fff has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0a: iomem range 0xfee00000-0xfee00fff could not be reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0b: ioport range 0xa00-0xa0f has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0d: iomem range 0xe0000000-0xefffffff could not be reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0e: iomem range 0x0-0x9ffff could not be reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0e: iomem range 0xc0000-0xcffff could not be reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0e: iomem range 0xe0000-0xfffff could not be reserved Feb 3 14:51:33 oss2 kernel: pnp: 00:0e: iomem range 0x100000-0xbfffffff could not be reserved Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:01.0 Feb 3 14:51:33 oss2 kernel: IO window: e000-efff Feb 3 14:51:33 oss2 kernel: MEM window: fbd00000-fbefffff Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:03.0 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: fbc00000-fbcfffff Feb 3 14:51:33 oss2 kernel: PREFETCH window 0x00000000fa800000-0x00000000faffffff Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:05.0 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: fbb00000-fbbfffff Feb 3 14:51:33 oss2 kernel: PREFETCH window 0x00000000fa000000-0x00000000fa7fffff Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:07.0 Feb 3 14:51:33 oss2 kernel: IO window: d000-dfff Feb 3 14:51:33 oss2 kernel: MEM window: fba00000-fbafffff Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:04:00.0 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: disabled. Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:04:00.2 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: disabled. Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:09.0 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: disabled. Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:1c.0 Feb 3 14:51:33 oss2 kernel: IO window: disabled. Feb 3 14:51:33 oss2 kernel: MEM window: disabled. Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:1c.4 Feb 3 14:51:33 oss2 kernel: IO window: c000-cfff Feb 3 14:51:33 oss2 kernel: MEM window: fb900000-fb9fffff Feb 3 14:51:33 oss2 kernel: PREFETCH window: disabled. Feb 3 14:51:33 oss2 kernel: PCI: Bridge: 0000:00:1e.0 Feb 3 14:51:33 oss2 kernel: IO window: b000-bfff Feb 3 14:51:33 oss2 kernel: MEM window: fb800000-fb8fffff Feb 3 14:51:33 oss2 kernel: PREFETCH window 0x00000000d8000000-0x00000000dfffffff Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:01.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:03.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:05.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:07.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:09.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:04:00.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:04:00.2 to 64 Feb 3 14:51:33 oss2 kernel: GSI 16 sharing vector 0xB1 and IRQ 16 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 17 (level, low) -> IRQ 177 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1c.0 to 64 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 17 (level, low) -> IRQ 177 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1c.4 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1e.0 to 64 Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 2 Feb 3 14:51:33 oss2 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Feb 3 14:51:33 oss2 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Feb 3 14:51:33 oss2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Feb 3 14:51:33 oss2 kernel: TCP: Hash tables configured (established 262144 bind 65536) Feb 3 14:51:33 oss2 kernel: TCP reno registered Feb 3 14:51:33 oss2 kernel: audit: initializing netlink socket (disabled) Feb 3 14:51:33 oss2 kernel: type=2000 audit(1328305815.184:1): initialized Feb 3 14:51:33 oss2 kernel: Total HugeTLB memory allocated, 0 Feb 3 14:51:33 oss2 kernel: VFS: Disk quotas dquot_6.5.1 Feb 3 14:51:33 oss2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Feb 3 14:51:33 oss2 kernel: SELinux: Registering netfilter hooks Feb 3 14:51:33 oss2 kernel: Initializing Cryptographic API Feb 3 14:51:33 oss2 kernel: alg: No test for crc32c (crc32c-generic) Feb 3 14:51:33 oss2 kernel: ksign: Installing public key data Feb 3 14:51:33 oss2 kernel: Loading keyring Feb 3 14:51:33 oss2 kernel: io scheduler noop registered Feb 3 14:51:33 oss2 kernel: io scheduler anticipatory registered Feb 3 14:51:33 oss2 kernel: io scheduler deadline registered (default) Feb 3 14:51:33 oss2 kernel: io scheduler cfq registered Feb 3 14:51:33 oss2 kernel: Boot video device is 0000:01:01.0 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:01.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:03.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:05.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:07.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:09.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1c.0 to 64 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1c.4 to 64 Feb 3 14:51:33 oss2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Feb 3 14:51:33 oss2 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707] Feb 3 14:51:33 oss2 kernel: ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707] Feb 3 14:51:33 oss2 kernel: Monitor-Mwait will be used to enter C-3 state Feb 3 14:51:33 oss2 kernel: Monitor-Mwait will be used to enter C-3 state Feb 3 14:51:33 oss2 kernel: ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU4 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU5 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU6 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU7 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU8 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU9 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU10 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU11 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU12 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU13 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU14 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU15 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU16 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU17 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU18 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU19 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU20 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU21 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU22 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: ACPI: CPU23 (power states: C1[C1] C2[C3] C3[C3]) Feb 3 14:51:33 oss2 kernel: Real Time Clock Driver v1.12ac Feb 3 14:51:33 oss2 kernel: hpet_resources: 0xfed00000 is busy Feb 3 14:51:33 oss2 kernel: Non-volatile memory driver v1.2 Feb 3 14:51:33 oss2 kernel: Linux agpgart interface v0.101 (c) Dave Jones Feb 3 14:51:33 oss2 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Feb 3 14:51:33 oss2 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Feb 3 14:51:33 oss2 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Feb 3 14:51:33 oss2 kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Feb 3 14:51:33 oss2 kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Feb 3 14:51:33 oss2 kernel: brd: module loaded Feb 3 14:51:33 oss2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Feb 3 14:51:33 oss2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Feb 3 14:51:33 oss2 kernel: JMB368: IDE controller at PCI slot 0000:02:00.0 Feb 3 14:51:33 oss2 kernel: GSI 17 sharing vector 0x3A and IRQ 17 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 58 Feb 3 14:51:33 oss2 kernel: JMB368: chipset revision 0 Feb 3 14:51:33 oss2 kernel: JMB368: 100% native mode on irq 58 Feb 3 14:51:33 oss2 kernel: ide2: BM-DMA at 0xc480-0xc487, BIOS settings: hde:pio, hdf:pio Feb 3 14:51:33 oss2 kernel: ide3: BM-DMA at 0xc488-0xc48f, BIOS settings: hdg:pio, hdh:pio Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide2... Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide3... Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide0... Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide1... Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide2... Feb 3 14:51:33 oss2 kernel: Probing IDE interface ide3... Feb 3 14:51:33 oss2 kernel: ide-floppy driver 0.99.newide Feb 3 14:51:33 oss2 kernel: usbcore: registered new driver hiddev Feb 3 14:51:33 oss2 kernel: usbcore: registered new driver usbhid Feb 3 14:51:33 oss2 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Feb 3 14:51:33 oss2 kernel: PNP: No PS/2 controller found. Probing ports directly. Feb 3 14:51:33 oss2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Feb 3 14:51:33 oss2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Feb 3 14:51:33 oss2 kernel: mice: PS/2 mouse device common for all mice Feb 3 14:51:33 oss2 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 Feb 3 14:51:33 oss2 kernel: md: bitmap version 4.39 Feb 3 14:51:33 oss2 kernel: TCP bic registered Feb 3 14:51:33 oss2 kernel: Initializing IPsec netlink socket Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 1 Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 17 Feb 3 14:51:33 oss2 kernel: ACPI: (supports S0 S1 S4 S5) Feb 3 14:51:33 oss2 kernel: Initalizing network drop monitor service Feb 3 14:51:33 oss2 kernel: Freeing unused kernel memory: 224k freed Feb 3 14:51:33 oss2 kernel: Write protecting the kernel read-only data: 521k Feb 3 14:51:33 oss2 kernel: GSI 18 sharing vector 0x42 and IRQ 18 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 66 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1a.7 to 64 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1a.7: EHCI Host Controller Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1a.7: debug port 1 Feb 3 14:51:33 oss2 kernel: PCI: cache line size of 32 is not supported by device 0000:00:1a.7 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1a.7: irq 66, io mem 0xfb7f4000 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Feb 3 14:51:33 oss2 kernel: usb usb1: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 1-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 1-0:1.0: 6 ports detected Feb 3 14:51:33 oss2 kernel: GSI 19 sharing vector 0x4A and IRQ 19 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 74 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1d.7 to 64 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1d.7: debug port 1 Feb 3 14:51:33 oss2 kernel: PCI: cache line size of 32 is not supported by device 0000:00:1d.7 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1d.7: irq 74, io mem 0xfb7f6000 Feb 3 14:51:33 oss2 kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 Feb 3 14:51:33 oss2 kernel: usb usb2: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 2-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 2-0:1.0: 6 ports detected Feb 3 14:51:33 oss2 kernel: ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI) Feb 3 14:51:33 oss2 kernel: USB Universal Host Controller Interface driver v3.0 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 58 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1a.0 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.0: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.0: irq 58, io base 0x00009400 Feb 3 14:51:33 oss2 kernel: usb usb3: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 3-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 3-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 2 Feb 3 14:51:33 oss2 kernel: GSI 21 sharing vector 0x52 and IRQ 21 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 82 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1a.1 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.1: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.1: irq 82, io base 0x00009480 Feb 3 14:51:33 oss2 kernel: usb usb4: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: usb 1-5: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input0 Feb 3 14:51:33 oss2 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Feb 3 14:51:33 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input1 Feb 3 14:51:33 oss2 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Feb 3 14:51:33 oss2 kernel: hub 4-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 4-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: GSI 22 sharing vector 0x5A and IRQ 22 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 90 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1a.2 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.2: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1a.2: irq 90, io base 0x00009800 Feb 3 14:51:33 oss2 kernel: usb usb5: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 5-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 5-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 74 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1d.0 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.0: irq 74, io base 0x00009880 Feb 3 14:51:33 oss2 kernel: usb usb6: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 6-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 6-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 90 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1d.1 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.1: irq 90, io base 0x00009c00 Feb 3 14:51:33 oss2 kernel: usb usb7: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 7-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 7-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 66 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1d.2 to 64 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8 Feb 3 14:51:33 oss2 kernel: uhci_hcd 0000:00:1d.2: irq 66, io base 0x0000a000 Feb 3 14:51:33 oss2 kernel: usb usb8: configuration #1 chosen from 1 choice Feb 3 14:51:33 oss2 kernel: hub 8-0:1.0: USB hub found Feb 3 14:51:33 oss2 kernel: hub 8-0:1.0: 2 ports detected Feb 3 14:51:33 oss2 kernel: SCSI subsystem initialized Feb 3 14:51:33 oss2 kernel: megasas: 00.00.04.31-RH1 Tues. June. 15 14:13:02 EST 2010 Feb 3 14:51:33 oss2 kernel: megasas: 0x1000:0x0079:0x15d9:0x0700: bus 7:slot 0:func 0 Feb 3 14:51:33 oss2 kernel: GSI 23 sharing vector 0x62 and IRQ 23 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:07:00.0[A] -> GSI 30 (level, low) -> IRQ 98 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:07:00.0 to 64 Feb 3 14:51:33 oss2 kernel: Feb 3 14:51:33 oss2 kernel: gen2: instance->base_addr = fba9c000<6>megasas: FW now in Ready state Feb 3 14:51:33 oss2 kernel: megasas_init_mfi: fw_support_ieee=0<6>scsi0 : LSI SAS based MegaRAID driver Feb 3 14:51:33 oss2 kernel: Vendor: SEAGATE Model: ST3300657SS Rev: 0008 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: Vendor: SEAGATE Model: ST3300657SS Rev: 0008 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: Vendor: SMC Model: SMC2108 Rev: 2.12 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: SCSI device sda: 583983104 512-byte hdwr sectors (298999 MB) Feb 3 14:51:33 oss2 kernel: sda: Write Protect is off Feb 3 14:51:33 oss2 kernel: sda: Mode Sense: 1f 00 00 08 Feb 3 14:51:33 oss2 kernel: SCSI device sda: drive cache: write back Feb 3 14:51:33 oss2 kernel: SCSI device sda: 583983104 512-byte hdwr sectors (298999 MB) Feb 3 14:51:33 oss2 kernel: sda: Write Protect is off Feb 3 14:51:33 oss2 kernel: sda: Mode Sense: 1f 00 00 08 Feb 3 14:51:33 oss2 kernel: SCSI device sda: drive cache: write back Feb 3 14:51:33 oss2 kernel: sda: sda1 sda2 Feb 3 14:51:33 oss2 kernel: sd 0:2:0:0: Attached scsi disk sda Feb 3 14:51:33 oss2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Feb 3 14:51:33 oss2 kernel: libata version 3.00 loaded. Feb 3 14:51:33 oss2 kernel: ahci 0000:00:1f.2: version 3.0 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 90 Feb 3 14:51:33 oss2 kernel: ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode Feb 3 14:51:33 oss2 kernel: ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:00:1f.2 to 64 Feb 3 14:51:33 oss2 kernel: scsi1 : ahci Feb 3 14:51:33 oss2 kernel: scsi2 : ahci Feb 3 14:51:33 oss2 kernel: scsi3 : ahci Feb 3 14:51:33 oss2 kernel: scsi4 : ahci Feb 3 14:51:33 oss2 kernel: scsi5 : ahci Feb 3 14:51:33 oss2 kernel: scsi6 : ahci Feb 3 14:51:33 oss2 kernel: ata1: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe100 irq 106 Feb 3 14:51:33 oss2 kernel: ata2: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe180 irq 106 Feb 3 14:51:33 oss2 kernel: ata3: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe200 irq 106 Feb 3 14:51:33 oss2 kernel: ata4: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe280 irq 106 Feb 3 14:51:33 oss2 kernel: ata5: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe300 irq 106 Feb 3 14:51:33 oss2 kernel: ata6: SATA max UDMA/133 abar m2048@0xfb7fe000 port 0xfb7fe380 irq 106 Feb 3 14:51:33 oss2 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) Feb 3 14:51:33 oss2 kernel: ata1.00: ATAPI: MATSHITADVD-RAM UJ8A0AS, 1.00, max UDMA/100, ATAPI AN Feb 3 14:51:33 oss2 kernel: ata1.00: configured for UDMA/100 Feb 3 14:51:33 oss2 kernel: ata2: SATA link down (SStatus 0 SControl 300) Feb 3 14:51:33 oss2 kernel: ata3: SATA link down (SStatus 0 SControl 300) Feb 3 14:51:33 oss2 kernel: ata4: SATA link down (SStatus 0 SControl 300) Feb 3 14:51:33 oss2 kernel: ata5: SATA link down (SStatus 0 SControl 300) Feb 3 14:51:33 oss2 kernel: ata6: SATA link down (SStatus 0 SControl 300) Feb 3 14:51:33 oss2 kernel: Vendor: MATSHITA Model: DVD-RAM UJ8A0AS Rev: 1.00 Feb 3 14:51:33 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: Initializing USB Mass Storage driver... Feb 3 14:51:33 oss2 kernel: scsi7 : SCSI emulation for USB Mass Storage devices Feb 3 14:51:33 oss2 kernel: usb-storage: device found at 2 Feb 3 14:51:33 oss2 kernel: usb-storage: waiting for device to settle before scanning Feb 3 14:51:33 oss2 kernel: scsi8 : SCSI emulation for USB Mass Storage devices Feb 3 14:51:33 oss2 kernel: usb-storage: device found at 2 Feb 3 14:51:33 oss2 kernel: usbcore: registered new driver usb-storage Feb 3 14:51:33 oss2 kernel: USB Mass Storage support registered. Feb 3 14:51:33 oss2 kernel: usb-storage: waiting for device to settle before scanning Feb 3 14:51:33 oss2 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Feb 3 14:51:33 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03<5> Vendor: PepperC Model: Virtual Disc 2 Rev: 0.01 Feb 3 14:51:33 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Feb 3 14:51:33 oss2 kernel: usb-storage: device scan complete Feb 3 14:51:33 oss2 kernel: Feb 3 14:51:33 oss2 kernel: usb-storage: device scan complete Feb 3 14:51:33 oss2 kernel: device-mapper: uevent: version 1.0.3 Feb 3 14:51:33 oss2 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com Feb 3 14:51:33 oss2 kernel: device-mapper: dm-raid45: initialized v0.2594l Feb 3 14:51:33 oss2 kernel: kjournald starting. Commit interval 5 seconds Feb 3 14:51:33 oss2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 3 14:51:33 oss2 kernel: SELinux: Disabled at runtime. Feb 3 14:51:33 oss2 kernel: SELinux: Unregistering netfilter hooks Feb 3 14:51:33 oss2 kernel: type=1404 audit(1328305851.825:2): selinux=0 auid=4294967295 ses=4294967295 Feb 3 14:51:33 oss2 kernel: input: PC Speaker as /class/input/input2 Feb 3 14:51:33 oss2 kernel: dca service started, version 1.8 Feb 3 14:51:33 oss2 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0 Feb 3 14:51:33 oss2 kernel: scsi 1:0:0:0: Attached scsi generic sg1 type 5 Feb 3 14:51:33 oss2 kernel: scsi 8:0:0:0: Attached scsi generic sg2 type 5 Feb 3 14:51:33 oss2 kernel: scsi 7:0:0:0: Attached scsi generic sg3 type 5 Feb 3 14:51:33 oss2 kernel: EDAC MC: Ver: 2.0.1 Jun 23 2011 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 66 Feb 3 14:51:33 oss2 kernel: 802.1Q VLAN Support v1.8 Ben Greear Feb 3 14:51:33 oss2 kernel: All bugs added by David S. Miller Feb 3 14:51:33 oss2 kernel: EDAC i7core: Driver loaded. Feb 3 14:51:33 oss2 kernel: Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2-1 Feb 3 14:51:33 oss2 kernel: Copyright (c) 2007-2009 Intel Corporation. Feb 3 14:51:33 oss2 kernel: GSI 24 sharing vector 0x72 and IRQ 24 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:0a:00.0[A] -> GSI 28 (level, low) -> IRQ 114 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:0a:00.0 to 64 Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.0: 0 vfs allocated Feb 3 14:51:33 oss2 kernel: sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray Feb 3 14:51:33 oss2 kernel: Uniform CD-ROM driver Revision: 3.20 Feb 3 14:51:33 oss2 kernel: sr 1:0:0:0: Attached scsi CD-ROM sr0 Feb 3 14:51:33 oss2 kernel: sr1: scsi-1 drive Feb 3 14:51:33 oss2 kernel: sr 8:0:0:0: Attached scsi CD-ROM sr1 Feb 3 14:51:33 oss2 kernel: sr2: scsi-1 drive Feb 3 14:51:33 oss2 kernel: sr 7:0:0:0: Attached scsi CD-ROM sr2 Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.0: Intel(R) Gigabit Ethernet Network Connection Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:25:90:2b:0d:22 Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.0: eth0: PBA No: 0010ff-0ff Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Feb 3 14:51:33 oss2 kernel: GSI 25 sharing vector 0xAA and IRQ 25 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:0a:00.1[B] -> GSI 40 (level, low) -> IRQ 170 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:0a:00.1 to 64 Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.1: 0 vfs allocated Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.1: Intel(R) Gigabit Ethernet Network Connection Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:25:90:2b:0d:23 Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.1: eth1: PBA No: 0010ff-0ff Feb 3 14:51:33 oss2 kernel: igb 0000:0a:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s) Feb 3 14:51:33 oss2 kernel: floppy0: no floppy controllers found Feb 3 14:51:33 oss2 kernel: work still pending Feb 3 14:51:33 oss2 kernel: lp: driver loaded but no devices found Feb 3 14:51:33 oss2 kernel: ACPI: Power Button (FF) [PWRF] Feb 3 14:51:33 oss2 kernel: ACPI: Power Button (CM) [PWRB] Feb 3 14:51:33 oss2 kernel: ACPI: Mapper loaded Feb 3 14:51:33 oss2 kernel: dell-wmi: No known WMI GUID found Feb 3 14:51:33 oss2 kernel: md: Autodetecting RAID arrays. Feb 3 14:51:33 oss2 kernel: md: autorun ... Feb 3 14:51:33 oss2 kernel: md: ... autorun DONE. Feb 3 14:51:33 oss2 kernel: device-mapper: multipath: version 1.0.6 loaded Feb 3 14:51:33 oss2 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded Feb 3 14:51:33 oss2 kernel: device-mapper: table: 253:5: multipath: error getting device Feb 3 14:51:33 oss2 kernel: device-mapper: ioctl: error adding target to table Feb 3 14:51:33 oss2 kernel: device-mapper: table: 253:5: multipath: error getting device Feb 3 14:51:33 oss2 kernel: device-mapper: ioctl: error adding target to table Feb 3 14:51:33 oss2 kernel: loop: loaded (max 8 devices) Feb 3 14:51:33 oss2 kernel: EXT3 FS on sda1, internal journal Feb 3 14:51:33 oss2 kernel: kjournald starting. Commit interval 5 seconds Feb 3 14:51:33 oss2 kernel: EXT3 FS on dm-0, internal journal Feb 3 14:51:33 oss2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 3 14:51:33 oss2 kernel: kjournald starting. Commit interval 5 seconds Feb 3 14:51:33 oss2 kernel: EXT3 FS on dm-1, internal journal Feb 3 14:51:33 oss2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 3 14:51:33 oss2 kernel: kjournald starting. Commit interval 5 seconds Feb 3 14:51:33 oss2 kernel: EXT3 FS on dm-2, internal journal Feb 3 14:51:33 oss2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 3 14:51:33 oss2 kernel: kjournald starting. Commit interval 5 seconds Feb 3 14:51:33 oss2 kernel: EXT3 FS on dm-3, internal journal Feb 3 14:51:33 oss2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Feb 3 14:51:33 oss2 kernel: Adding 2097144k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1 across:2097144k Feb 3 14:51:33 oss2 kernel: IA-32 Microcode Update Driver: v1.14a Feb 3 14:51:33 oss2 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011) Feb 3 14:51:33 oss2 kernel: mlx4_core: Initializing 0000:09:00.0 Feb 3 14:51:33 oss2 kernel: GSI 26 sharing vector 0xE2 and IRQ 26 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:09:00.0[A] -> GSI 24 (level, low) -> IRQ 226 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:09:00.0 to 64 Feb 3 14:51:33 oss2 kernel: mlx4_core: Initializing 0000:08:00.0 Feb 3 14:51:33 oss2 kernel: GSI 27 sharing vector 0x34 and IRQ 27 Feb 3 14:51:33 oss2 kernel: ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 26 (level, low) -> IRQ 52 Feb 3 14:51:33 oss2 kernel: PCI: Setting latency timer of device 0000:08:00.0 to 64 Feb 3 14:51:33 oss2 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.3 (Jan 2011) Feb 3 14:51:33 oss2 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011) Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 10 Feb 3 14:51:33 oss2 kernel: lo: Disabled Privacy Extensions Feb 3 14:51:33 oss2 kernel: IPv6 over IPv4 tunneling driver Feb 3 14:51:33 oss2 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready Feb 3 14:51:33 oss2 kernel: ib0: enabling connected mode will cause multicast packet drops Feb 3 14:51:33 oss2 kernel: ib0: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: ib0: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: ib1: enabling connected mode will cause multicast packet drops Feb 3 14:51:33 oss2 kernel: ib1: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: ib1: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: ib2: enabling connected mode will cause multicast packet drops Feb 3 14:51:33 oss2 kernel: ib2: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: ib2: mtu > 2044 will cause multicast packet drops. Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 27 Feb 3 14:51:33 oss2 kernel: NET: Registered protocol family 28 Feb 3 14:51:33 oss2 kernel: device-mapper: table: 253:5: multipath: error getting device Feb 3 14:51:33 oss2 kernel: device-mapper: ioctl: error adding target to table Feb 3 14:51:33 oss2 kernel: device-mapper: table: 253:5: multipath: error getting device Feb 3 14:51:33 oss2 kernel: device-mapper: ioctl: error adding target to table Feb 3 14:51:33 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 11 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 11 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_1 Feb 3 14:51:33 oss2 kernel: eth0: no IPv6 routers present Feb 3 14:51:33 oss2 kernel: host9: ib_srp: new target: id_ext 189d4c0003c90200 ioc_guid 189d4c0003c90200 pkey ffff service_id 189d4c0003c90200 dgid fe80:0000:0000:0000:0002:c903:004c:9d1a Feb 3 14:51:33 oss2 kernel: scsi9 : SRP.T10:189D4C0003C90200 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sd 9:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Feb 3 14:51:33 oss2 kernel: sdb: Unit Not Ready, sense: Feb 3 14:51:33 oss2 kernel: : Current: sense key: Unit Attention Feb 3 14:51:33 oss2 kernel: Add. Sense: Reported luns data has changed Feb 3 14:51:33 oss2 kernel: Feb 3 14:51:33 oss2 kernel: sdb : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdb: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdb: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdb: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdb: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdb : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdb: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdb: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdb: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdb: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdb: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:0: Attached scsi disk sdb Feb 3 14:51:33 oss2 kernel: sd 9:0:0:0: Attached scsi generic sg4 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdc : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdc: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdc: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdc: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdc: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdc : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdc: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdc: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdc: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdc: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdc: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:1: Attached scsi disk sdc Feb 3 14:51:33 oss2 kernel: sd 9:0:0:1: Attached scsi generic sg5 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdd : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdd: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdd: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdd: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdd: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdd : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdd: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdd: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdd: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdd: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdd: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:2: Attached scsi disk sdd Feb 3 14:51:33 oss2 kernel: sd 9:0:0:2: Attached scsi generic sg6 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sde : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sde: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sde: Write Protect is off Feb 3 14:51:33 oss2 kernel: sde: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sde: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sde : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sde: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sde: Write Protect is off Feb 3 14:51:33 oss2 kernel: sde: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sde: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sde: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:3: Attached scsi disk sde Feb 3 14:51:33 oss2 kernel: sd 9:0:0:3: Attached scsi generic sg7 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdf : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdf: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdf: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdf: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdf: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdf : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdf: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdf: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdf: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdf: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdf: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:4: Attached scsi disk sdf Feb 3 14:51:33 oss2 kernel: sd 9:0:0:4: Attached scsi generic sg8 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdg : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdg: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdg: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdg: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdg: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdg : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdg: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdg: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdg: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdg: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdg: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:5: Attached scsi disk sdg Feb 3 14:51:33 oss2 kernel: sd 9:0:0:5: Attached scsi generic sg9 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdh : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdh: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdh: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdh: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdh: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdh : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdh: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdh: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdh: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdh: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdh: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:6: Attached scsi disk sdh Feb 3 14:51:33 oss2 kernel: sd 9:0:0:6: Attached scsi generic sg10 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdi : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdi: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdi: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdi: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdi: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdi : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdi: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdi: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdi: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdi: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdi: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:7: Attached scsi disk sdi Feb 3 14:51:33 oss2 kernel: sd 9:0:0:7: Attached scsi generic sg11 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdj : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdj: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdj: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdj: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdj: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdj : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdj: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdj: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdj: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdj: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdj: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:8: Attached scsi disk sdj Feb 3 14:51:33 oss2 kernel: sd 9:0:0:8: Attached scsi generic sg12 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdk : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdk: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdk: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdk: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdk: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdk : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdk: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdk: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdk: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdk: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdk: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:9: Attached scsi disk sdk Feb 3 14:51:33 oss2 kernel: sd 9:0:0:9: Attached scsi generic sg13 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdl : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdl: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdl: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdl: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdl: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdl : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdl: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdl: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdl: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdl: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdl: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:10: Attached scsi disk sdl Feb 3 14:51:33 oss2 kernel: sd 9:0:0:10: Attached scsi generic sg14 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdm : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdm: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdm: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdm: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdm: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdm : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdm: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdm: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdm: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdm: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdm: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:11: Attached scsi disk sdm Feb 3 14:51:33 oss2 kernel: sd 9:0:0:11: Attached scsi generic sg15 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdn : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdn: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdn: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdn: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdn: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdn : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdn: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdn: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdn: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdn: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdn: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:12: Attached scsi disk sdn Feb 3 14:51:33 oss2 kernel: sd 9:0:0:12: Attached scsi generic sg16 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdo : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdo: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdo: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdo: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdo: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdo : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdo: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdo: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdo: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdo: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdo: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:13: Attached scsi disk sdo Feb 3 14:51:33 oss2 kernel: sd 9:0:0:13: Attached scsi generic sg17 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdp : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdp: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdp: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdp: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdp: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdp : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdp: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdp: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdp: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdp: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdp: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:14: Attached scsi disk sdp Feb 3 14:51:33 oss2 kernel: sd 9:0:0:14: Attached scsi generic sg18 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdq : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdq: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdq: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdq: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdq: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdq : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdq: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdq: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdq: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdq: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdq: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:15: Attached scsi disk sdq Feb 3 14:51:33 oss2 kernel: sd 9:0:0:15: Attached scsi generic sg19 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdr : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdr: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdr: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdr: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdr: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdr : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdr: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdr: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdr: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdr: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdr: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:16: Attached scsi disk sdr Feb 3 14:51:33 oss2 kernel: sd 9:0:0:16: Attached scsi generic sg20 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sds : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sds: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sds: Write Protect is off Feb 3 14:51:33 oss2 kernel: sds: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sds: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sds : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sds: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sds: Write Protect is off Feb 3 14:51:33 oss2 kernel: sds: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sds: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sds: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:17: Attached scsi disk sds Feb 3 14:51:33 oss2 kernel: sd 9:0:0:17: Attached scsi generic sg21 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdt : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdt: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdt: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdt: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdt: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdt : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdt: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdt: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdt: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdt: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdt: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:18: Attached scsi disk sdt Feb 3 14:51:33 oss2 kernel: sd 9:0:0:18: Attached scsi generic sg22 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdu : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdu: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdu: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdu: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdu: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdu : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdu: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdu: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdu: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdu: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdu: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:19: Attached scsi disk sdu Feb 3 14:51:33 oss2 kernel: sd 9:0:0:19: Attached scsi generic sg23 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdv : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdv: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdv: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdv: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdv: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdv : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdv: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdv: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdv: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdv: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdv: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:20: Attached scsi disk sdv Feb 3 14:51:33 oss2 kernel: sd 9:0:0:20: Attached scsi generic sg24 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdw : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdw: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdw: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdw: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdw: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdw : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdw: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdw: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdw: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdw: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdw: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:21: Attached scsi disk sdw Feb 3 14:51:33 oss2 kernel: sd 9:0:0:21: Attached scsi generic sg25 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdx : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdx: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdx: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdx: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdx: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdx : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdx: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdx: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdx: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdx: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdx: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:22: Attached scsi disk sdx Feb 3 14:51:33 oss2 kernel: sd 9:0:0:22: Attached scsi generic sg26 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdy : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdy: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdy: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdy: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdy: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdy : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdy: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdy: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdy: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdy: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdy: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:23: Attached scsi disk sdy Feb 3 14:51:33 oss2 kernel: sd 9:0:0:23: Attached scsi generic sg27 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdz : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdz: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdz: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdz: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdz: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdz : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdz: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdz: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdz: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdz: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdz: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 9:0:0:24: Attached scsi disk sdz Feb 3 14:51:33 oss2 kernel: sd 9:0:0:24: Attached scsi generic sg28 type 0 Feb 3 14:51:33 oss2 kernel: host10: ib_srp: new target: id_ext 4c9e4c0003c90200 ioc_guid 4c9e4c0003c90200 pkey ffff service_id 4c9e4c0003c90200 dgid fe80:0000:0000:0000:0002:c903:004c:9e4e Feb 3 14:51:33 oss2 kernel: scsi10 : SRP.T10:4C9E4C0003C90200 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sd 10:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Feb 3 14:51:33 oss2 kernel: sdaa: Unit Not Ready, sense: Feb 3 14:51:33 oss2 kernel: : Current: sense key: Unit Attention Feb 3 14:51:33 oss2 kernel: Add. Sense: Reported luns data has changed Feb 3 14:51:33 oss2 kernel: Feb 3 14:51:33 oss2 kernel: sdaa : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaa: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaa: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaa: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaa: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaa : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaa: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaa: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaa: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaa: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaa: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:0: Attached scsi disk sdaa Feb 3 14:51:33 oss2 kernel: sd 10:0:0:0: Attached scsi generic sg29 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdab : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdab: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdab: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdab: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdab: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdab : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdab: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdab: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdab: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdab: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdab: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:1: Attached scsi disk sdab Feb 3 14:51:33 oss2 kernel: sd 10:0:0:1: Attached scsi generic sg30 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdac : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdac: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdac: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdac: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdac: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdac : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdac: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdac: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdac: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdac: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdac: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:2: Attached scsi disk sdac Feb 3 14:51:33 oss2 kernel: sd 10:0:0:2: Attached scsi generic sg31 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdad : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdad: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdad: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdad: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdad: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdad : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdad: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdad: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdad: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdad: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdad: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:3: Attached scsi disk sdad Feb 3 14:51:33 oss2 kernel: sd 10:0:0:3: Attached scsi generic sg32 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdae : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdae: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdae: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdae: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdae: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdae : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdae: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdae: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdae: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdae: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdae: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:4: Attached scsi disk sdae Feb 3 14:51:33 oss2 kernel: sd 10:0:0:4: Attached scsi generic sg33 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdaf : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaf: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaf: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaf: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaf: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaf : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaf: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaf: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaf: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaf: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaf: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:5: Attached scsi disk sdaf Feb 3 14:51:33 oss2 kernel: sd 10:0:0:5: Attached scsi generic sg34 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdag : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdag: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdag: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdag: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdag: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdag : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdag: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdag: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdag: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdag: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdag: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:6: Attached scsi disk sdag Feb 3 14:51:33 oss2 kernel: sd 10:0:0:6: Attached scsi generic sg35 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdah : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdah: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdah: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdah: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdah: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdah : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdah: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdah: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdah: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdah: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdah: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:7: Attached scsi disk sdah Feb 3 14:51:33 oss2 kernel: sd 10:0:0:7: Attached scsi generic sg36 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdai : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdai: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdai: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdai: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdai: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdai : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdai: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdai: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdai: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdai: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdai: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:8: Attached scsi disk sdai Feb 3 14:51:33 oss2 kernel: sd 10:0:0:8: Attached scsi generic sg37 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdaj : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaj: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaj: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaj: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaj: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaj : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaj: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaj: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaj: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaj: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaj: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:9: Attached scsi disk sdaj Feb 3 14:51:33 oss2 kernel: sd 10:0:0:9: Attached scsi generic sg38 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdak : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdak: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdak: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdak: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdak: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdak : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdak: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdak: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdak: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdak: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdak: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:10: Attached scsi disk sdak Feb 3 14:51:33 oss2 kernel: sd 10:0:0:10: Attached scsi generic sg39 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdal : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdal: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdal: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdal: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdal: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdal : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdal: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdal: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdal: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdal: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdal: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:11: Attached scsi disk sdal Feb 3 14:51:33 oss2 kernel: sd 10:0:0:11: Attached scsi generic sg40 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdam : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdam: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdam: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdam: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdam: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdam : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdam: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdam: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdam: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdam: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdam: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:12: Attached scsi disk sdam Feb 3 14:51:33 oss2 kernel: sd 10:0:0:12: Attached scsi generic sg41 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdan : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdan: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdan: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdan: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdan: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdan : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdan: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdan: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdan: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdan: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdan: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:13: Attached scsi disk sdan Feb 3 14:51:33 oss2 kernel: sd 10:0:0:13: Attached scsi generic sg42 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdao : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdao: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdao: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdao: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdao: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdao : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdao: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdao: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdao: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdao: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdao: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:14: Attached scsi disk sdao Feb 3 14:51:33 oss2 kernel: sd 10:0:0:14: Attached scsi generic sg43 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdap : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdap: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdap: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdap: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdap: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdap : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdap: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdap: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdap: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdap: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdap: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:15: Attached scsi disk sdap Feb 3 14:51:33 oss2 kernel: sd 10:0:0:15: Attached scsi generic sg44 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdaq : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaq: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaq: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaq: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaq: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaq : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaq: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaq: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaq: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaq: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaq: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:16: Attached scsi disk sdaq Feb 3 14:51:33 oss2 kernel: sd 10:0:0:16: Attached scsi generic sg45 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdar : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdar: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdar: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdar: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdar: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdar : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdar: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdar: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdar: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdar: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdar: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:17: Attached scsi disk sdar Feb 3 14:51:33 oss2 kernel: sd 10:0:0:17: Attached scsi generic sg46 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdas : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdas: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdas: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdas: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdas: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdas : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdas: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdas: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdas: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdas: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdas: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:18: Attached scsi disk sdas Feb 3 14:51:33 oss2 kernel: sd 10:0:0:18: Attached scsi generic sg47 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdat : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdat: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdat: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdat: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdat: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdat : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdat: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdat: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdat: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdat: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdat: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:19: Attached scsi disk sdat Feb 3 14:51:33 oss2 kernel: sd 10:0:0:19: Attached scsi generic sg48 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdau : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdau: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdau: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdau: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdau: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdau : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdau: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdau: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdau: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdau: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdau: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:20: Attached scsi disk sdau Feb 3 14:51:33 oss2 kernel: sd 10:0:0:20: Attached scsi generic sg49 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdav : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdav: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdav: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdav: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdav: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdav : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdav: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdav: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdav: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdav: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdav: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:21: Attached scsi disk sdav Feb 3 14:51:33 oss2 kernel: sd 10:0:0:21: Attached scsi generic sg50 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdaw : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaw: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaw: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaw: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaw: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaw : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdaw: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdaw: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdaw: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdaw: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdaw: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:22: Attached scsi disk sdaw Feb 3 14:51:33 oss2 kernel: sd 10:0:0:22: Attached scsi generic sg51 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sdax : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdax: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdax: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdax: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdax: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdax : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sdax: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sdax: Write Protect is off Feb 3 14:51:33 oss2 kernel: sdax: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sdax: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sdax: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:23: Attached scsi disk sdax Feb 3 14:51:33 oss2 kernel: sd 10:0:0:23: Attached scsi generic sg52 type 0 Feb 3 14:51:33 oss2 kernel: Vendor: SGI Model: DD6A-IS16K-10000 Rev: 1.03 Feb 3 14:51:33 oss2 kernel: Type: Direct-Access ANSI SCSI revision: 05 Feb 3 14:51:33 oss2 kernel: sday : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sday: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sday: Write Protect is off Feb 3 14:51:33 oss2 kernel: sday: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sday: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sday : very big device. try to use READ CAPACITY(16). Feb 3 14:51:33 oss2 kernel: SCSI device sday: 30584864768 512-byte hdwr sectors (15659451 MB) Feb 3 14:51:33 oss2 kernel: sday: Write Protect is off Feb 3 14:51:33 oss2 kernel: sday: Mode Sense: 6f 00 10 08 Feb 3 14:51:33 oss2 kernel: SCSI device sday: drive cache: write back w/ FUA Feb 3 14:51:33 oss2 kernel: sday: unknown partition table Feb 3 14:51:33 oss2 kernel: sd 10:0:0:24: Attached scsi disk sday Feb 3 14:51:33 oss2 kernel: sd 10:0:0:24: Attached scsi generic sg53 type 0 Feb 3 14:51:33 oss2 kernel: ipmi message handler version 39.1 Feb 3 14:51:33 oss2 kernel: IPMI System Interface driver. Feb 3 14:51:33 oss2 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 Feb 3 14:51:33 oss2 kernel: ipmi: Found new BMC (man_id: 0x0028c5, prod_id: 0x0004, dev_id: 0x22) Feb 3 14:51:33 oss2 kernel: IPMI kcs interface initialized Feb 3 14:51:34 oss2 kernel: ipmi device interface Feb 3 14:51:40 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 3 14:51:40 oss2 kernel: ib_srp: ASYNC event= 11 on device= mlx4_0 Feb 3 14:51:41 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Feb 3 14:51:55 oss2 kernel: ib_srp: ASYNC event= 12 on device= mlx4_0 Feb 3 14:52:01 oss2 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Feb 3 14:52:12 oss2 kernel: ib0: no IPv6 routers present Feb 3 15:01:08 oss2 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Feb 3 15:01:08 oss2 kernel: Lustre: Lustre Version: 1.8.6 Feb 3 15:01:08 oss2 kernel: Lustre: Build Version: jenkins-wc1--PRISTINE-2.6.18-238.12.1.el5_lustre.gce5e033 Feb 3 15:01:08 oss2 kernel: Lustre: Listener bound to ib0:192.168.64.8:987:mlx4_0 Feb 3 15:01:08 oss2 kernel: Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Feb 3 15:01:08 oss2 kernel: Lustre: Added LNI 192.168.64.8@o2ib [8/64/0/180] Feb 3 15:01:08 oss2 kernel: Lustre: Lustre Client File System; http://www.lustre.org/ Feb 3 15:01:08 oss2 kernel: init dynlocks cache Feb 3 15:01:08 oss2 kernel: ldiskfs created from ext4-2.6-rhel5 Feb 3 15:01:08 oss2 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Feb 3 15:01:08 oss2 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Feb 3 15:01:08 oss2 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Feb 3 15:01:08 oss2 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Feb 3 15:01:08 oss2 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Feb 3 15:01:08 oss2 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Feb 3 15:01:08 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Feb 3 15:01:09 oss2 kernel: fsfilt_ldiskfs: no version for "ldiskfs_iget" found: kernel tainted. Feb 3 15:01:09 oss2 kernel: Lustre: Filtering OBD driver; http://wiki.whamcloud.com/ Feb 3 15:01:09 oss2 kernel: Lustre: 13456:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0005, 1 recoverable clients, 0 delayed clients, last_rcvd 17180031680 Feb 3 15:01:09 oss2 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Feb 3 15:01:09 oss2 kernel: Lustre: lustre-OST0005: Now serving lustre-OST0005 on /dev/mapper/ost_lustre_5 with recovery enabled Feb 3 15:01:09 oss2 kernel: Lustre: lustre-OST0005: Will be in recovery for at least 5:00, or until 1 client reconnects Feb 3 15:01:09 oss2 kernel: Lustre: 13456:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0005: quotaon failed because quota files don't exist, please run quotacheck firstly Feb 3 15:01:09 oss2 kernel: Lustre: 13493:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0001, 1 recoverable clients, 0 delayed clients, last_rcvd 17180142096 Feb 3 15:01:09 oss2 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Feb 3 15:01:09 oss2 kernel: Lustre: lustre-OST0001: Now serving lustre-OST0001 on /dev/mapper/ost_lustre_1 with recovery enabled Feb 3 15:01:09 oss2 kernel: Lustre: lustre-OST0001: Will be in recovery for at least 5:00, or until 1 client reconnects Feb 3 15:01:09 oss2 kernel: Lustre: 13493:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0001: quotaon failed because quota files don't exist, please run quotacheck firstly Feb 3 15:01:10 oss2 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Feb 3 15:01:10 oss2 kernel: LDISKFS-fs (dm-18): mounted filesystem with ordered data mode Feb 3 15:01:10 oss2 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Feb 3 15:01:10 oss2 kernel: JBD: barrier-based sync failed on dm-18-8 - disabling barriers Feb 3 15:01:10 oss2 kernel: LDISKFS-fs (dm-14): mounted filesystem with ordered data mode Feb 3 15:01:10 oss2 kernel: LDISKFS-fs (dm-18): mounted filesystem with ordered data mode Feb 3 15:01:11 oss2 kernel: Lustre: 13819:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0009, 1 recoverable clients, 0 delayed clients, last_rcvd 17180039931 Feb 3 15:01:11 oss2 kernel: JBD: barrier-based sync failed on dm-14-8 - disabling barriers Feb 3 15:01:11 oss2 kernel: Lustre: lustre-OST0009: Now serving lustre-OST0009 on /dev/mapper/ost_lustre_9 with recovery enabled Feb 3 15:01:11 oss2 kernel: Lustre: lustre-OST0009: Will be in recovery for at least 5:00, or until 1 client reconnects Feb 3 15:01:11 oss2 kernel: Lustre: 13819:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0009: quotaon failed because quota files don't exist, please run quotacheck firstly Feb 3 15:01:11 oss2 kernel: JBD: barrier-based sync failed on dm-18-8 - disabling barriers Feb 3 15:01:12 oss2 kernel: LDISKFS-fs (dm-22): mounted filesystem with ordered data mode Feb 3 15:01:12 oss2 kernel: JBD: barrier-based sync failed on dm-22-8 - disabling barriers Feb 3 15:01:12 oss2 kernel: LDISKFS-fs (dm-22): mounted filesystem with ordered data mode Feb 3 15:01:12 oss2 kernel: Lustre: 14023:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0011, 1 recoverable clients, 0 delayed clients, last_rcvd 17180042553 Feb 3 15:01:12 oss2 kernel: Lustre: 14023:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Feb 3 15:01:12 oss2 kernel: JBD: barrier-based sync failed on dm-22-8 - disabling barriers Feb 3 15:01:13 oss2 kernel: Lustre: lustre-OST0011: Now serving lustre-OST0011 on /dev/mapper/ost_lustre_17 with recovery enabled Feb 3 15:01:13 oss2 kernel: Lustre: Skipped 1 previous similar message Feb 3 15:01:13 oss2 kernel: Lustre: lustre-OST0011: Will be in recovery for at least 5:00, or until 1 client reconnects Feb 3 15:01:13 oss2 kernel: Lustre: Skipped 1 previous similar message Feb 3 15:01:13 oss2 kernel: Lustre: 14023:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0011: quotaon failed because quota files don't exist, please run quotacheck firstly Feb 3 15:01:13 oss2 kernel: Lustre: 14023:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Feb 3 15:01:14 oss2 kernel: LDISKFS-fs (dm-26): mounted filesystem with ordered data mode Feb 3 15:01:14 oss2 kernel: JBD: barrier-based sync failed on dm-26-8 - disabling barriers Feb 3 15:01:14 oss2 kernel: LDISKFS-fs (dm-26): mounted filesystem with ordered data mode Feb 3 15:01:14 oss2 kernel: JBD: barrier-based sync failed on dm-26-8 - disabling barriers Feb 3 15:01:32 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Feb 3 15:01:32 oss2 kernel: LustreError: 13171:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063e8d0800 x1392830273552427/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1328306592 ref 1 fl Interpret:/0/0 rc -19/0 Feb 3 15:01:32 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Feb 3 15:01:32 oss2 kernel: LustreError: Skipped 1 previous similar message Feb 3 15:01:32 oss2 kernel: LustreError: 13171:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3dc27c00 x1392830273552431/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1328306592 ref 1 fl Interpret:/0/0 rc -19/0 Feb 3 15:01:32 oss2 kernel: LustreError: 13171:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Feb 3 15:01:32 oss2 kernel: Lustre: lustre-OST0015: Recovery period over after 0:01, of 1 clients 1 recovered and 0 were evicted. Feb 3 15:01:32 oss2 kernel: Lustre: lustre-OST0015: sending delayed replies to recovered clients Feb 3 15:01:32 oss2 kernel: Lustre: lustre-OST0015: received MDS connection from 192.168.64.5@o2ib Feb 3 15:01:32 oss2 kernel: Lustre: 13175:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0015: deleting orphan objects from 195 to 225, orphan objids won't be reused any more. Feb 3 15:01:38 oss2 kernel: Lustre: lustre-OST0001: Recovery period over after 0:01, of 1 clients 1 recovered and 0 were evicted. Feb 3 15:01:38 oss2 kernel: Lustre: lustre-OST000d: sending delayed replies to recovered clients Feb 3 15:01:38 oss2 kernel: Lustre: Skipped 1 previous similar message Feb 3 15:01:38 oss2 kernel: Lustre: lustre-OST0011: received MDS connection from 192.168.64.5@o2ib Feb 3 15:01:38 oss2 kernel: Lustre: 13191:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0011: deleting orphan objects from 195 to 225, orphan objids won't be reused any more. Feb 4 07:50:47 oss2 kernel: Lustre: 13179:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: e11fbec1-b337-0137-53ab-62464ed2fb5c reconnecting Feb 4 07:50:47 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: e11fbec1-b337-0137-53ab-62464ed2fb5c reconnecting Feb 4 07:50:47 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Feb 4 07:50:47 oss2 kernel: Lustre: 13179:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Feb 6 12:05:20 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 3af80ccd-eae8-a015-5b4f-1f892db9c657 (at 192.168.2.13@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Feb 6 12:13:07 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c013e8f5-af68-8edb-6ddb-a56ba3b0a20a (at 192.168.2.50@o2ib) in 228 seconds. I think it's dead, and I am evicting it. Feb 6 12:13:07 oss2 kernel: Lustre: Skipped 23 previous similar messages Feb 6 12:52:23 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: e11fbec1-b337-0137-53ab-62464ed2fb5c reconnecting Feb 6 12:52:23 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Feb 6 12:52:23 oss2 kernel: Lustre: 13220:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: e11fbec1-b337-0137-53ab-62464ed2fb5c reconnecting Feb 6 12:52:23 oss2 kernel: Lustre: 13220:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Feb 7 12:20:06 oss2 kernel: Lustre: 13198:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: e11fbec1-b337-0137-53ab-62464ed2fb5c reconnecting Feb 7 12:20:06 oss2 kernel: Lustre: 13198:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Feb 7 15:47:27 oss2 kernel: Lustre: 13229:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: 33515a07-3f9a-20d4-6c95-b5de30705118 reconnecting Feb 7 15:47:27 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Feb 7 15:47:27 oss2 kernel: LustreError: Skipped 4 previous similar messages Feb 7 15:47:27 oss2 kernel: LustreError: 13245:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810603378800 x1392842298002494/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1328654947 ref 1 fl Interpret:/0/0 rc -19/0 Feb 7 15:47:27 oss2 kernel: LustreError: 13245:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Feb 7 15:47:33 oss2 kernel: Lustre: 13244:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 33515a07-3f9a-20d4-6c95-b5de30705118 reconnecting Feb 7 15:47:33 oss2 kernel: Lustre: 13244:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Feb 8 14:05:51 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Feb 8 14:05:51 oss2 kernel: LustreError: Skipped 6 previous similar messages Feb 8 14:05:51 oss2 kernel: LustreError: 13270:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3e2c4400 x1392842372626288/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1328735251 ref 1 fl Interpret:/0/0 rc -19/0 Feb 8 14:05:51 oss2 kernel: LustreError: 13270:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Feb 8 14:05:53 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 8a350d34-d43e-ab51-3903-1280e5a97abb (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Feb 8 14:05:53 oss2 kernel: Lustre: Skipped 503 previous similar messages Feb 8 14:15:28 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 8a350d34-d43e-ab51-3903-1280e5a97abb (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Feb 8 14:15:28 oss2 kernel: Lustre: Skipped 5 previous similar messages Feb 10 11:53:26 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 11:56:11 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:05:54 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:12:18 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:15:42 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:17:33 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:22:51 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:25:17 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 12:29:32 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 14:54:56 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 9d08ad8a-e492-6805-df6d-3640c2eedf15 (at 192.168.2.10@o2ib) in 257 seconds. I think it's dead, and I am evicting it. Feb 10 14:54:56 oss2 kernel: Lustre: Skipped 5 previous similar messages Feb 10 15:06:33 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 16:42:20 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 16:52:15 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 10 17:07:38 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Feb 23 17:16:13 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Mar 7 14:30:00 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Mar 9 19:30:38 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 5cfa1775-ff54-5ec6-f38e-799ffc7b8d18 (at 192.168.11.26@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Mar 9 19:30:38 oss2 kernel: Lustre: Skipped 59 previous similar messages Mar 12 17:53:30 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 1bcc0815-1116-559f-d0ec-4a0ac525a420 (at 192.168.11.18@o2ib) in 237 seconds. I think it's dead, and I am evicting it. Mar 12 17:53:30 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:36:53 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 53d14a62-b178-4ba4-dc11-acdedc4ba362 (at 192.168.11.18@o2ib) in 235 seconds. I think it's dead, and I am evicting it. Mar 13 14:36:53 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:08:05 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client c5ccfcb1-092e-4efe-35c5-521bcb5ac7c6 (at 192.168.11.18@o2ib) in 242 seconds. I think it's dead, and I am evicting it. Mar 13 16:08:05 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 13 17:34:45 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 9ed191b8-75ec-982a-2ba2-00cc56fc1d62 (at 192.168.11.18@o2ib) in 244 seconds. I think it's dead, and I am evicting it. Mar 13 17:34:45 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:45:13 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client f5da903d-8a35-a56c-3717-05ee508a39d4 (at 192.168.11.18@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Mar 14 08:45:13 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:45:13 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client f5da903d-8a35-a56c-3717-05ee508a39d4 (at 192.168.11.18@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Mar 16 16:20:36 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 5fb33028-6735-6ac4-cf6f-66d58f6b7c33 (at 192.168.20.2@o2ib) in 248 seconds. I think it's dead, and I am evicting it. Mar 16 16:20:36 oss2 kernel: Lustre: Skipped 4 previous similar messages Mar 19 16:31:04 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 19e475e7-1843-c712-c800-3157176d246c (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Mar 19 16:31:04 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 20 00:32:43 oss2 kernel: Lustre: Service thread pid 13220 was inactive for 0.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 20 00:32:43 oss2 kernel: Pid: 13220, comm: ll_ost_51 Mar 20 00:32:43 oss2 kernel: Mar 20 00:32:43 oss2 kernel: Call Trace: Mar 20 00:32:43 oss2 kernel: Lustre: Service thread pid 13220 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 20 00:32:43 oss2 kernel: [] lc_watchdog_disable+0xcb/0x130 [libcfs] Mar 20 00:32:43 oss2 kernel: [] ptlrpc_wait_event+0x1f5/0x310 [ptlrpc] Mar 20 00:32:43 oss2 kernel: [] default_wake_function+0x0/0xe Mar 20 00:32:43 oss2 kernel: [] ptlrpc_main+0x3db/0x1120 [ptlrpc] Mar 20 00:32:43 oss2 kernel: [] child_rip+0xa/0x11 Mar 20 00:32:43 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 20 00:32:43 oss2 kernel: [] child_rip+0x0/0x11 Mar 20 00:32:43 oss2 kernel: Mar 20 07:14:23 oss2 kernel: Lustre: Service thread pid 13234 was inactive for 0.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 20 07:14:23 oss2 kernel: Pid: 13234, comm: ll_ost_65 Mar 20 07:14:23 oss2 kernel: Mar 20 07:14:23 oss2 kernel: Call Trace: Mar 20 07:14:23 oss2 kernel: [] lcw_cb+0x0/0x460 [libcfs] Mar 20 07:14:23 oss2 kernel: [] libcfs_debug_dumpstack+0x51/0x60 [libcfs] Mar 20 07:14:23 oss2 kernel: [] lcw_cb+0x33c/0x460 [libcfs] Mar 20 07:14:23 oss2 kernel: [] run_timer_softirq+0x193/0x241 Mar 20 07:14:23 oss2 kernel: [] __do_softirq+0x89/0x133 Mar 20 07:14:23 oss2 kernel: [] call_softirq+0x1c/0x28 Mar 20 07:14:23 oss2 kernel: [] do_softirq+0x2c/0x7d Mar 20 07:14:23 oss2 kernel: [] apic_timer_interrupt+0x66/0x6c Mar 20 07:14:23 oss2 kernel: [] __mod_timer+0x28/0x10f Mar 20 07:14:23 oss2 kernel: [] lc_watchdog_touch+0xe9/0x150 [libcfs] Mar 20 07:14:23 oss2 kernel: [] remove_wait_queue+0x1c/0x2c Mar 20 07:14:23 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 20 07:14:23 oss2 kernel: [] default_wake_function+0x0/0xe Mar 20 07:14:23 oss2 kernel: [] ptlrpc_main+0x3db/0x1120 [ptlrpc] Mar 20 07:14:23 oss2 kernel: [] child_rip+0xa/0x11 Mar 20 07:14:23 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 20 07:14:23 oss2 kernel: [] child_rip+0x0/0x11 Mar 20 07:14:23 oss2 kernel: Mar 20 07:14:23 oss2 kernel: Lustre: Service thread pid 13234 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 20 13:02:18 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 81b46a13-b2b6-a869-c917-6df1bfedcf03 (at 192.168.11.18@o2ib) in 275 seconds. I think it's dead, and I am evicting it. Mar 20 13:02:18 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 21 14:40:39 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 7b480f1b-1d0b-a650-2e16-ff08b79dfdbd (at 192.168.11.18@o2ib) in 241 seconds. I think it's dead, and I am evicting it. Mar 21 14:40:39 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 21 18:28:45 oss2 kernel: Lustre: Service thread pid 13252 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 21 18:28:45 oss2 kernel: Lustre: Service thread pid 13252 was inactive for 0.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 21 18:28:45 oss2 kernel: Pid: 13252, comm: ll_ost_83 Mar 21 18:28:45 oss2 kernel: Mar 21 18:28:45 oss2 kernel: Call Trace: Mar 21 18:28:45 oss2 kernel: [] ptlrpc_wait_event+0x1f5/0x310 [ptlrpc] Mar 21 18:28:45 oss2 kernel: [] default_wake_function+0x0/0xe Mar 21 18:28:45 oss2 kernel: [] ptlrpc_main+0x3db/0x1120 [ptlrpc] Mar 21 18:28:45 oss2 kernel: [] child_rip+0xa/0x11 Mar 21 18:28:45 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 21 18:28:45 oss2 kernel: [] child_rip+0x0/0x11 Mar 21 18:28:45 oss2 kernel: Mar 29 18:27:30 oss2 kernel: Lustre: 13178:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Mar 29 18:27:30 oss2 kernel: Lustre: 13178:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Mar 29 18:29:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Mar 29 18:29:28 oss2 kernel: LustreError: Skipped 6 previous similar messages Mar 29 18:29:28 oss2 kernel: LustreError: 13272:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810619057400 x1397268322760387/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333067468 ref 1 fl Interpret:/0/0 rc -19/0 Mar 29 18:29:28 oss2 kernel: LustreError: 13272:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Mar 29 18:29:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Mar 29 18:29:28 oss2 kernel: LustreError: Skipped 3 previous similar messages Mar 29 18:29:28 oss2 kernel: LustreError: 13250:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3e4a4c00 x1397268322760403/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333067468 ref 1 fl Interpret:/0/0 rc -19/0 Mar 29 18:29:28 oss2 kernel: LustreError: 13250:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Mar 29 18:29:39 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client fdfe6d24-762a-2357-5e80-1f62efef7d0b (at 192.168.11.18@o2ib) in 237 seconds. I think it's dead, and I am evicting it. Mar 29 18:29:39 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 29 21:26:33 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 394016b7-f08e-bd97-e2a3-1c060daae9e4 (at 192.168.11.19@o2ib) in 243 seconds. I think it's dead, and I am evicting it. Mar 29 21:26:33 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 29 23:37:34 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 42626e21-60b0-5ddd-50ad-11ed15e39b63 (at 192.168.11.19@o2ib) in 247 seconds. I think it's dead, and I am evicting it. Mar 29 23:37:34 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 08:13:49 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client da52a8fd-43b9-c54b-0a9b-ce156943645d (at 192.168.11.19@o2ib) in 243 seconds. I think it's dead, and I am evicting it. Mar 30 08:13:49 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 09:24:14 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 39eb7aaf-3735-c38b-6732-66f9a31f0481 (at 192.168.11.19@o2ib) in 240 seconds. I think it's dead, and I am evicting it. Mar 30 09:24:14 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 10:33:13 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 69be30dc-6085-4993-2b34-3490b0ea8aa0 (at 192.168.11.19@o2ib) in 241 seconds. I think it's dead, and I am evicting it. Mar 30 10:33:13 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 15:26:19 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 224a094a-2be9-e7e2-efcd-c0407461d4b0 (at 192.168.11.19@o2ib) in 231 seconds. I think it's dead, and I am evicting it. Mar 30 15:26:19 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 15:55:54 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c996ed5c-6a08-16d8-a6d2-a2be18c12890 (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Mar 30 15:55:54 oss2 kernel: Lustre: Skipped 4 previous similar messages Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(o2iblnd_cb.c:2912:kiblnd_check_txs()) Timed out tx: tx_queue, 1 seconds Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(o2iblnd_cb.c:2975:kiblnd_check_conns()) Timed out RDMA with 192.168.11.19@o2ib (51) Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 4, status -103, desc ffff8102c94f8000 Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 4, status -103, desc ffff8106ea202000 Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 2, status -103, desc ffff8106ea202000 Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 4, status -103, desc ffff8106ff84a000 Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 4, status -103, desc ffff810b401ca000 Mar 30 20:37:03 oss2 kernel: LustreError: 12752:0:(events.c:381:server_bulk_callback()) event type 4, status -103, desc ffff810336c98000 Mar 30 20:37:03 oss2 kernel: LustreError: 4361:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff810226428800 x1397917116282728/t0 o4->b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 2 to 0 dl 1333161455 ref 1 fl Interpret:/0/0 rc 0/0 Mar 30 20:37:03 oss2 kernel: Lustre: 4361:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0009: ignoring bulk IO comm error with b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 30 20:39:31 oss2 kernel: Lustre: Service thread pid 13397 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 30 20:39:31 oss2 kernel: Pid: 13397, comm: ll_ost_io_98 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: Call Trace: Mar 30 20:39:31 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Mar 30 20:39:31 oss2 kernel: [] schedule_timeout+0x8a/0xad Mar 30 20:39:31 oss2 kernel: [] process_timeout+0x0/0x5 Mar 30 20:39:31 oss2 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Mar 30 20:39:31 oss2 kernel: [] default_wake_function+0x0/0xe Mar 30 20:39:31 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Mar 30 20:39:31 oss2 kernel: [] __next_cpu+0x19/0x28 Mar 30 20:39:31 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] __wake_up_common+0x3e/0x68 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0xa/0x11 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0x0/0x11 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1333161571.13397 Mar 30 20:39:31 oss2 kernel: Lustre: Service thread pid 4433 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 30 20:39:31 oss2 kernel: Pid: 4433, comm: ll_ost_io_342 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: Call Trace: Mar 30 20:39:31 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Mar 30 20:39:31 oss2 kernel: [] lock_timer_base+0x1b/0x3c Mar 30 20:39:31 oss2 kernel: [] __mod_timer+0x100/0x10f Mar 30 20:39:31 oss2 kernel: [] schedule_timeout+0x8a/0xad Mar 30 20:39:31 oss2 kernel: [] process_timeout+0x0/0x5 Mar 30 20:39:31 oss2 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Mar 30 20:39:31 oss2 kernel: [] default_wake_function+0x0/0xe Mar 30 20:39:31 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Mar 30 20:39:31 oss2 kernel: [] __next_cpu+0x19/0x28 Mar 30 20:39:31 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] __wake_up_common+0x3e/0x68 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0xa/0x11 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0x0/0x11 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1333161571.4433 Mar 30 20:39:31 oss2 kernel: Pid: 4499, comm: ll_ost_io_407 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: Call Trace: Mar 30 20:39:31 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Mar 30 20:39:31 oss2 kernel: [] lock_timer_base+0x1b/0x3c Mar 30 20:39:31 oss2 kernel: [] __mod_timer+0x100/0x10f Mar 30 20:39:31 oss2 kernel: [] schedule_timeout+0x8a/0xad Mar 30 20:39:31 oss2 kernel: [] process_timeout+0x0/0x5 Mar 30 20:39:31 oss2 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Mar 30 20:39:31 oss2 kernel: [] default_wake_function+0x0/0xe Mar 30 20:39:31 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Mar 30 20:39:31 oss2 kernel: [] __next_cpu+0x19/0x28 Mar 30 20:39:31 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] __wake_up_common+0x3e/0x68 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0xa/0x11 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0x0/0x11 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: Pid: 13360, comm: ll_ost_io_61 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: Call Trace: Mar 30 20:39:31 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Mar 30 20:39:31 oss2 kernel: [] schedule_timeout+0x8a/0xad Mar 30 20:39:31 oss2 kernel: [] process_timeout+0x0/0x5 Mar 30 20:39:31 oss2 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Mar 30 20:39:31 oss2 kernel: [] default_wake_function+0x0/0xe Mar 30 20:39:31 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Mar 30 20:39:31 oss2 kernel: [] __next_cpu+0x19/0x28 Mar 30 20:39:31 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] __wake_up_common+0x3e/0x68 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0xa/0x11 Mar 30 20:39:31 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 30 20:39:31 oss2 kernel: [] child_rip+0x0/0x11 Mar 30 20:39:31 oss2 kernel: Mar 30 20:39:31 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1333161571.13360 Mar 30 20:39:31 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1333161571.4499 Mar 30 20:40:03 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client b662025e-4546-072d-99f3-135bdf99929e (at 192.168.11.19@o2ib) in 247 seconds. I think it's dead, and I am evicting it. Mar 30 20:40:03 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 30 20:40:04 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client b662025e-4546-072d-99f3-135bdf99929e (at 192.168.11.19@o2ib) in 233 seconds. I think it's dead, and I am evicting it. Mar 30 20:40:04 oss2 kernel: Lustre: Skipped 3 previous similar messages Mar 30 20:40:05 oss2 kernel: LustreError: 13397:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff810c371dac00 x1397917116282725/t0 o3->b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID:0/0 lens 448/400 e 4 to 0 dl 1333161767 ref 1 fl Interpret:/0/0 rc 0/0 Mar 30 20:40:05 oss2 kernel: Lustre: 13397:0:(ost_handler.c:887:ost_brw_read()) lustre-OST0011: ignoring bulk IO comm error with b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 30 20:40:05 oss2 kernel: Lustre: Service thread pid 13397 completed after 234.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 30 20:40:05 oss2 kernel: LustreError: 13360:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff810695f08000 x1397917116282730/t0 o3->b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID:0/0 lens 448/400 e 4 to 0 dl 1333161767 ref 1 fl Interpret:/0/0 rc 0/0 Mar 30 20:40:05 oss2 kernel: LustreError: 13360:0:(ost_handler.c:825:ost_brw_read()) Skipped 1 previous similar message Mar 30 20:40:05 oss2 kernel: Lustre: 13360:0:(ost_handler.c:887:ost_brw_read()) lustre-OST0011: ignoring bulk IO comm error with b662025e-4546-072d-99f3-135bdf99929e@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 30 20:40:05 oss2 kernel: Lustre: 13360:0:(ost_handler.c:887:ost_brw_read()) Skipped 1 previous similar message Mar 30 20:40:05 oss2 kernel: Lustre: Service thread pid 13360 completed after 234.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 30 20:40:05 oss2 kernel: Lustre: Skipped 1 previous similar message Mar 31 04:51:34 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 5bbd1970-fe00-c0de-b113-846f5f2970b5 (at 192.168.11.19@o2ib) in 238 seconds. I think it's dead, and I am evicting it. Mar 31 04:51:34 oss2 kernel: Lustre: Skipped 1 previous similar message Mar 31 05:11:59 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 0c692319-3dc8-056c-f46a-be3ff933f2c1 (at 192.168.11.19@o2ib) in 239 seconds. I think it's dead, and I am evicting it. Mar 31 05:11:59 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 06:51:34 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 065be2eb-9852-6c67-e48d-4e842786a83e (at 192.168.11.19@o2ib) in 232 seconds. I think it's dead, and I am evicting it. Mar 31 06:51:34 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 09:17:49 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 9b10ebaa-d144-b352-9c4f-3f2a7d69d17f (at 192.168.11.19@o2ib) in 231 seconds. I think it's dead, and I am evicting it. Mar 31 09:17:49 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 09:42:09 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 320355d5-f198-afa5-f3b3-c0e056473a69 (at 192.168.11.19@o2ib) in 233 seconds. I think it's dead, and I am evicting it. Mar 31 09:42:09 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 13:07:09 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105834a0000 Mar 31 13:07:09 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105c639e000 Mar 31 13:07:09 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105834a0000 Mar 31 13:07:09 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105c639e000 Mar 31 13:07:09 oss2 kernel: LustreError: 4432:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81061a3fd000 x1397980550928662/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1333220840 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8100acefa000 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8100acefa000 Mar 31 13:07:09 oss2 kernel: Lustre: 4432:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0015: ignoring bulk IO comm error with 11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 31 13:07:09 oss2 kernel: LustreError: 12742:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8104a5608000 Mar 31 13:07:09 oss2 kernel: LustreError: 12742:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8104a5608000 Mar 31 13:07:09 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108d95b8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108d95b8000 Mar 31 13:07:09 oss2 kernel: LustreError: 4543:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81069d8c2000 x1397980550928665/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1333220840 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:07:09 oss2 kernel: LustreError: 4543:0:(ost_handler.c:1073:ost_brw_write()) Skipped 3 previous similar messages Mar 31 13:07:09 oss2 kernel: Lustre: 4543:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0015: ignoring bulk IO comm error with 11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 31 13:07:09 oss2 kernel: Lustre: 4543:0:(ost_handler.c:1224:ost_brw_write()) Skipped 3 previous similar messages Mar 31 13:07:09 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810a6e414000 Mar 31 13:07:09 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810a6e414000 Mar 31 13:07:09 oss2 kernel: Lustre: 13242:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 11bc233d-7358-5b35-9b08-f2a0fe337e7c reconnecting Mar 31 13:07:09 oss2 kernel: Lustre: 13242:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Mar 31 13:07:09 oss2 kernel: Lustre: 13242:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 11bc233d-7358-5b35-9b08-f2a0fe337e7c@192.168.11.19@o2ib to 0xffff810604bde400; still busy with 18 active RPCs Mar 31 13:07:09 oss2 kernel: LustreError: 13242:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810611e13850 x1397980550928685/t0 o8->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1333220929 ref 1 fl Interpret:/0/0 rc -16/0 Mar 31 13:07:09 oss2 kernel: LustreError: 13242:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Mar 31 13:07:09 oss2 kernel: LustreError: 4587:0:(ost_handler.c:1078:ost_brw_write()) @@@ ptlrpc_bulk_get failed: rc -107 req@ffff81049a800c00 x1397980550928683/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333220835 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:07:09 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8107922c8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8101fb708000 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8101fb708000 Mar 31 13:07:09 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8107922c8000 Mar 31 13:07:09 oss2 kernel: LustreError: 7825:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8102d2f52800 x1397980550928667/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333220835 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:07:09 oss2 kernel: LustreError: 12736:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108adcfc000 Mar 31 13:07:09 oss2 kernel: LustreError: 12736:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108adcfc000 Mar 31 13:07:09 oss2 kernel: LustreError: 7803:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8108608cac00 x1397980550928669/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333220835 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:07:09 oss2 kernel: LustreError: 7803:0:(ost_handler.c:1064:ost_brw_write()) Skipped 1 previous similar message Mar 31 13:07:09 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108b88e8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108b88e8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105c639e000 Mar 31 13:07:09 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105c639e000 Mar 31 13:07:09 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810280132000 Mar 31 13:07:09 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810280132000 Mar 31 13:07:09 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810a58a00000 Mar 31 13:07:09 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810a58a00000 Mar 31 13:07:09 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8109b3966000 Mar 31 13:07:09 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8109b3966000 Mar 31 13:07:09 oss2 kernel: LustreError: 12728:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff81091d6de000 Mar 31 13:07:09 oss2 kernel: LustreError: 12728:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff81091d6de000 Mar 31 13:07:09 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff81067d6fc000 Mar 31 13:07:09 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff81067d6fc000 Mar 31 13:07:09 oss2 kernel: LustreError: 12747:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8101a51a8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12747:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8101a51a8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff81020db66000 Mar 31 13:07:09 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff81020db66000 Mar 31 13:07:09 oss2 kernel: LustreError: 12732:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8100a1c0c000 Mar 31 13:07:09 oss2 kernel: LustreError: 12732:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8100a1c0c000 Mar 31 13:07:09 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108d95b8000 Mar 31 13:07:09 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108d95b8000 Mar 31 13:07:10 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8107c9586000 Mar 31 13:07:10 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8107c9586000 Mar 31 13:07:10 oss2 kernel: LustreError: 12737:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8101b79e8000 Mar 31 13:07:10 oss2 kernel: LustreError: 12737:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8101b79e8000 Mar 31 13:07:10 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff81049ed04000 Mar 31 13:07:10 oss2 kernel: LustreError: 12743:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff81049ed04000 Mar 31 13:07:12 oss2 kernel: Lustre: 13252:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 11bc233d-7358-5b35-9b08-f2a0fe337e7c reconnecting Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8106051ba000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8106051ba000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108d95b8000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108d95b8000 Mar 31 13:20:49 oss2 kernel: LustreError: 4555:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff810227f77c00 x1397980550986502/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1333221661 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:20:49 oss2 kernel: LustreError: 4555:0:(ost_handler.c:1073:ost_brw_write()) Skipped 1 previous similar message Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: Lustre: 4555:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0015: ignoring bulk IO comm error with 11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 31 13:20:49 oss2 kernel: Lustre: 4555:0:(ost_handler.c:1224:ost_brw_write()) Skipped 19 previous similar messages Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108e9894000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108e9894000 Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810136582000 Mar 31 13:20:49 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105af40c000 Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810136582000 Mar 31 13:20:49 oss2 kernel: LustreError: 12729:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105af40c000 Mar 31 13:20:49 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105c639e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105c639e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810c31508000 Mar 31 13:20:49 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810c31508000 Mar 31 13:20:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810bce3b4c00 Mar 31 13:20:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810bce3b4c00 Mar 31 13:20:49 oss2 kernel: LustreError: 12732:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105a2928000 Mar 31 13:20:49 oss2 kernel: LustreError: 12732:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105a2928000 Mar 31 13:20:49 oss2 kernel: LustreError: 12739:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810981678000 Mar 31 13:20:49 oss2 kernel: LustreError: 12739:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810981678000 Mar 31 13:20:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810541c8d000 Mar 31 13:20:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810541c8d000 Mar 31 13:20:49 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8104fce60000 Mar 31 13:20:49 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8104fce60000 Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810136582000 Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810136582000 Mar 31 13:20:49 oss2 kernel: LustreError: 12736:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810c31508000 Mar 31 13:20:49 oss2 kernel: LustreError: 12736:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810c31508000 Mar 31 13:20:49 oss2 kernel: LustreError: 12749:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105c639e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12749:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105c639e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8107bed5e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810b04540000 Mar 31 13:20:49 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8107bed5e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810b04540000 Mar 31 13:20:49 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: LustreError: 13358:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8104f4e7c000 x1397980550986511/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333221655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:20:49 oss2 kernel: LustreError: 13308:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81060fe4bc50 x1397980550986518/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333221655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:20:49 oss2 kernel: LustreError: 12738:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: LustreError: 13308:0:(ost_handler.c:1073:ost_brw_write()) Skipped 15 previous similar messages Mar 31 13:20:49 oss2 kernel: Lustre: 13308:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0015: ignoring bulk IO comm error with 11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Mar 31 13:20:49 oss2 kernel: Lustre: 13308:0:(ost_handler.c:1224:ost_brw_write()) Skipped 16 previous similar messages Mar 31 13:20:49 oss2 kernel: LustreError: 12735:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8103dc4f2000 Mar 31 13:20:49 oss2 kernel: LustreError: 12735:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8103dc4f2000 Mar 31 13:20:49 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8108d95b8000 Mar 31 13:20:49 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8108d95b8000 Mar 31 13:20:49 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810018fb0000 Mar 31 13:20:49 oss2 kernel: LustreError: 12731:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810018fb0000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8103b4cd0000 Mar 31 13:20:49 oss2 kernel: LustreError: 12748:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8103b4cd0000 Mar 31 13:20:49 oss2 kernel: LustreError: 12747:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8106cc166000 Mar 31 13:20:49 oss2 kernel: LustreError: 12747:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8106cc166000 Mar 31 13:20:49 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810421a3a000 Mar 31 13:20:49 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810421a3a000 Mar 31 13:20:49 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105a2928000 Mar 31 13:20:49 oss2 kernel: LustreError: 12751:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105a2928000 Mar 31 13:20:49 oss2 kernel: LustreError: 12735:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8107b6c3e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12735:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8107b6c3e000 Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: Lustre: 13175:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 11bc233d-7358-5b35-9b08-f2a0fe337e7c reconnecting Mar 31 13:20:49 oss2 kernel: LustreError: 12750:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810440ad8000 Mar 31 13:20:49 oss2 kernel: Lustre: 13175:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 11bc233d-7358-5b35-9b08-f2a0fe337e7c@192.168.11.19@o2ib to 0xffff810604bde400; still busy with 6 active RPCs Mar 31 13:20:49 oss2 kernel: LustreError: 13175:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810611ee1c50 x1397980550986533/t0 o8->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1333221749 ref 1 fl Interpret:/0/0 rc -16/0 Mar 31 13:20:49 oss2 kernel: LustreError: 13426:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff81027e3ea000 x1397980550986529/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333221655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:20:49 oss2 kernel: LustreError: 13426:0:(ost_handler.c:1064:ost_brw_write()) Skipped 14 previous similar messages Mar 31 13:20:49 oss2 kernel: LustreError: 7834:0:(ost_handler.c:1078:ost_brw_write()) @@@ ptlrpc_bulk_get failed: rc -107 req@ffff81060fb3c850 x1397980550986532/t0 o4->11bc233d-7358-5b35-9b08-f2a0fe337e7c@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1333221655 ref 1 fl Interpret:/0/0 rc 0/0 Mar 31 13:20:49 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810202ca4000 Mar 31 13:20:49 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810202ca4000 Mar 31 13:20:49 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8101d4194000 Mar 31 13:20:49 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8101d4194000 Mar 31 13:20:49 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8104f034a000 Mar 31 13:20:49 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8104f034a000 Mar 31 13:20:51 oss2 kernel: Lustre: 13209:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 11bc233d-7358-5b35-9b08-f2a0fe337e7c reconnecting Mar 31 14:34:39 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 11bc233d-7358-5b35-9b08-f2a0fe337e7c (at 192.168.11.19@o2ib) in 231 seconds. I think it's dead, and I am evicting it. Mar 31 14:34:39 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 15:55:27 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 0d059d5b-7a6e-744b-cd03-0ceefd80077f (at 192.168.11.19@o2ib) in 237 seconds. I think it's dead, and I am evicting it. Mar 31 15:55:27 oss2 kernel: Lustre: Skipped 5 previous similar messages Mar 31 15:55:27 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 0d059d5b-7a6e-744b-cd03-0ceefd80077f (at 192.168.11.19@o2ib) in 241 seconds. I think it's dead, and I am evicting it. Mar 31 15:55:27 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:27:08 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 15:27:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190014 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 15:27:40 oss2 kernel: req@ffff8106072fb800 x1392830283190014/t0 o400->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 192/384 e 0 to 1 dl 1333315660 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:27:40 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.5@o2ib was lost; in progress operations using this service will fail. Apr 1 15:27:46 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190015 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:27:46 oss2 kernel: req@ffff81046347ec00 x1392830283190015/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333315666 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:27:53 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190016 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:27:53 oss2 kernel: req@ffff810153d45400 x1392830283190016/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333315673 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:28:07 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190018 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:28:07 oss2 kernel: req@ffff81061e80f000 x1392830283190018/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333315687 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:28:07 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 15:28:18 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 15:28:19 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 15:28:21 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190020 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:28:21 oss2 kernel: req@ffff810618c96000 x1392830283190020/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333315701 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:28:21 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 15:28:22 oss2 kernel: Lustre: 5332:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:22 oss2 kernel: Lustre: 13264:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 15:28:22 oss2 kernel: Lustre: 5332:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:22 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 15:28:22 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 1 15:28:22 oss2 kernel: LustreError: 13288:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810ad9049800 x1397284104399997/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315802 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:22 oss2 kernel: LustreError: 13288:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101ea3f2800 x1397284104400021/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315802 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:22 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:22 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 133 previous similar messages Apr 1 15:28:23 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:23 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 242 previous similar messages Apr 1 15:28:25 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:25 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 508 previous similar messages Apr 1 15:28:26 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 15:28:26 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 15:28:26 oss2 kernel: LustreError: 13169:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061140e850 x1397272694004595/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315806 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:26 oss2 kernel: LustreError: 13169:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Apr 1 15:28:29 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:29 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 962 previous similar messages Apr 1 15:28:29 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 15:28:29 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 12 previous similar messages Apr 1 15:28:29 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Apr 1 15:28:29 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 15:28:29 oss2 kernel: LustreError: 13223:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81050a61d800 x1397284104400117/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315809 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:29 oss2 kernel: LustreError: 13223:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 15:28:35 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190022 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:28:35 oss2 kernel: req@ffff8101281e5000 x1392830283190022/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333315715 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:28:35 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 15:28:35 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 15:28:35 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 15:28:35 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 15:28:35 oss2 kernel: Lustre: 13292:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 1 15:28:35 oss2 kernel: Lustre: 13292:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Apr 1 15:28:37 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.5@o2ib Apr 1 15:28:37 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 1846 previous similar messages Apr 1 15:28:38 oss2 kernel: Lustre: 13176:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 9d98c294-b076-0f80-e073-d2e0279f8b2e reconnecting Apr 1 15:28:38 oss2 kernel: Lustre: 13176:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 15:28:42 oss2 kernel: LustreError: 25375:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810abbffa400 x1392830283190027/t0 o501->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 264/432 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 15:28:42 oss2 kernel: Lustre: 25408:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0018, 5 recoverable clients, 0 delayed clients, last_rcvd 25770773628 Apr 1 15:28:42 oss2 kernel: Lustre: 25408:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Apr 1 15:28:42 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 15:28:42 oss2 kernel: Lustre: lustre-OST0018: Now serving lustre-OST0018 on /dev/mapper/ost_lustre_24 with recovery enabled Apr 1 15:28:42 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:28:42 oss2 kernel: Lustre: lustre-OST0018: Will be in recovery for at least 5:00, or until 5 clients reconnect Apr 1 15:28:42 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:28:42 oss2 kernel: Lustre: 25408:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0018: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 15:28:42 oss2 kernel: Lustre: 25408:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Apr 1 15:28:42 oss2 kernel: LustreError: 25117:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81061a3fcc00 x1392830283190039/t0 o253->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 4736/4928 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 15:28:42 oss2 kernel: LustreError: 25117:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 11 previous similar messages Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: Lustre: 25533:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0000, 5 recoverable clients, 0 delayed clients, last_rcvd 25770632120 Apr 1 15:28:43 oss2 kernel: Lustre: 25533:0:(filter.c:1001:filter_init_server_data()) Skipped 2 previous similar messages Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/mapper/ost_lustre_0 with recovery enabled Apr 1 15:28:43 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 15:28:43 oss2 kernel: Lustre: lustre-OST0000: Will be in recovery for at least 5:00, or until 5 clients reconnect Apr 1 15:28:43 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 15:28:43 oss2 kernel: Lustre: 25533:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0000: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 15:28:43 oss2 kernel: Lustre: 25533:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 2 previous similar messages Apr 1 15:28:43 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 15:28:43 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 15:28:43 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 15:28:43 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190045 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 0s ago has failed due to network error (6s prior to deadline). Apr 1 15:28:43 oss2 kernel: req@ffff8104a6134400 x1392830283190045/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333315729 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 15:28:43 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 15:28:43 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 15:28:43 oss2 kernel: LustreError: 25543:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8104a6134000 x1392830283190048/t0 o101->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 15:28:43 oss2 kernel: LustreError: 25543:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Apr 1 15:28:43 oss2 kernel: Lustre: 25753:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0008, 5 recoverable clients, 0 delayed clients, last_rcvd 25770741409 Apr 1 15:28:43 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 15:28:43 oss2 kernel: Lustre: lustre-OST0008: Now serving lustre-OST0008 on /dev/mapper/ost_lustre_8 with recovery enabled Apr 1 15:28:43 oss2 kernel: Lustre: lustre-OST0008: Will be in recovery for at least 5:00, or until 5 clients reconnect Apr 1 15:28:43 oss2 kernel: Lustre: 25753:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0008: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 15:28:44 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 15:28:44 oss2 kernel: Lustre: 13269:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: lustre-mdtlov_UUID reconnecting Apr 1 15:28:44 oss2 kernel: Lustre: 13276:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: lustre-mdtlov_UUID reconnecting Apr 1 15:28:44 oss2 kernel: Lustre: 13276:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 15:28:44 oss2 kernel: Lustre: 13269:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 15:28:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Apr 1 15:28:44 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 15:28:44 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b090c5800 x1392830278847751/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315824 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:44 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 15:28:44 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 15:28:44 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 15:28:44 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 15:28:44 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 15:28:44 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:28:45 oss2 kernel: Lustre: 13200:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0000: 4 recoverable clients remain Apr 1 15:28:45 oss2 kernel: Lustre: 13211:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 4 recoverable clients remain Apr 1 15:28:45 oss2 kernel: Lustre: 13211:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 3 previous similar messages Apr 1 15:28:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Apr 1 15:28:48 oss2 kernel: LustreError: Skipped 1 previous similar message Apr 1 15:28:48 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b578df000 x1397284104400178/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315828 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:28:48 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 1 15:28:48 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0008: 3 recoverable clients remain Apr 1 15:28:48 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 1 previous similar message Apr 1 15:28:49 oss2 kernel: Lustre: lustre-OST000d: received MDS connection from 192.168.64.5@o2ib Apr 1 15:28:49 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 1 15:28:49 oss2 kernel: Lustre: 13233:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST000d: deleting orphan objects from 234 to 257, orphan objids won't be reused any more. Apr 1 15:28:49 oss2 kernel: Lustre: 13233:0:(filter.c:3126:filter_destroy_precreated()) Skipped 4 previous similar messages Apr 1 15:28:51 oss2 kernel: LustreError: 25891:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8104ae310400 x1392830283190096/t0 o101->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 15:28:51 oss2 kernel: LustreError: 25891:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Apr 1 15:28:51 oss2 kernel: Lustre: 26107:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0010, 5 recoverable clients, 0 delayed clients, last_rcvd 25770630333 Apr 1 15:28:51 oss2 kernel: Lustre: 26107:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Apr 1 15:28:51 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 15:28:51 oss2 kernel: Lustre: lustre-OST0010: Now serving lustre-OST0010 on /dev/mapper/ost_lustre_16 with recovery enabled Apr 1 15:28:51 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:28:51 oss2 kernel: Lustre: lustre-OST0010: Will be in recovery for at least 5:00, or until 5 clients reconnect Apr 1 15:28:51 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 15:28:51 oss2 kernel: Lustre: 26107:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0010: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 15:28:51 oss2 kernel: Lustre: 26107:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Apr 1 15:28:55 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 15:29:04 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190097 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 15:29:04 oss2 kernel: req@ffff8104318a2000 x1392830283190097/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333315744 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 15:29:04 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Apr 1 15:29:17 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 15:29:17 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 15:29:19 oss2 kernel: Lustre: 13213:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 2 recoverable clients remain Apr 1 15:29:19 oss2 kernel: Lustre: 13213:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 7 previous similar messages Apr 1 15:29:20 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000c: lustre-mdtlov_UUID reconnecting Apr 1 15:29:20 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8 previous similar messages Apr 1 15:29:26 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 15:29:29 oss2 kernel: Lustre: 13209:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0010: 2 recoverable clients remain Apr 1 15:29:29 oss2 kernel: Lustre: 13209:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 10 previous similar messages Apr 1 15:29:36 oss2 kernel: Lustre: 13221:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0010: 1 recoverable clients remain Apr 1 15:30:10 oss2 kernel: Lustre: Failing over lustre-OST0000 Apr 1 15:30:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) Apr 1 15:30:10 oss2 kernel: LustreError: 13288:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8104c024e800 x1392830278848083/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315910 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:30:10 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0008: lustre-mdtlov_UUID reconnecting Apr 1 15:30:10 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Apr 1 15:30:10 oss2 kernel: Lustre: 13277:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000c: refuse reconnection from lustre-mdtlov_UUID@192.168.64.5@o2ib to 0xffff8105e63ab000; still busy with 1 active RPCs Apr 1 15:30:10 oss2 kernel: Lustre: 13277:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages Apr 1 15:30:11 oss2 kernel: Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Apr 1 15:30:11 oss2 kernel: Lustre: OST lustre-OST0000 has stopped. Apr 1 15:30:11 oss2 kernel: Lustre: server umount lustre-OST0000 complete Apr 1 15:30:11 oss2 kernel: Lustre: lustre-OST0018: shutting down for failover; client state will be preserved. Apr 1 15:30:11 oss2 kernel: Lustre: OST lustre-OST0018 has stopped. Apr 1 15:30:11 oss2 kernel: Lustre: server umount lustre-OST0018 complete Apr 1 15:30:12 oss2 kernel: Lustre: Failing over lustre-OST000c Apr 1 15:30:12 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 15:30:13 oss2 kernel: Lustre: lustre-OST0010: shutting down for failover; client state will be preserved. Apr 1 15:30:13 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 15:30:13 oss2 kernel: Lustre: OST lustre-OST0010 has stopped. Apr 1 15:30:13 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 15:30:13 oss2 kernel: Lustre: server umount lustre-OST0010 complete Apr 1 15:30:13 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 15:30:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Apr 1 15:30:24 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 1 15:30:25 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:813:target_handle_connect()) lustre-OST0009: received MDS connection from NID 192.168.64.6@o2ib, removing former export from NID 192.168.64.5@o2ib Apr 1 15:30:25 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:813:target_handle_connect()) Skipped 1 previous similar message Apr 1 15:30:25 oss2 kernel: Lustre: 13215:0:(ldlm_lib.c:813:target_handle_connect()) lustre-OST000d: received MDS connection from NID 192.168.64.6@o2ib, removing former export from NID 192.168.64.5@o2ib Apr 1 15:30:31 oss2 kernel: LustreError: 13253:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810aee688000 x1398082925232175/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315931 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:30:31 oss2 kernel: LustreError: 13253:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 15:30:31 oss2 kernel: Lustre: 13244:0:(ldlm_lib.c:813:target_handle_connect()) lustre-OST0015: received MDS connection from NID 192.168.64.6@o2ib, removing former export from NID 192.168.64.5@o2ib Apr 1 15:30:31 oss2 kernel: Lustre: 13244:0:(ldlm_lib.c:813:target_handle_connect()) Skipped 1 previous similar message Apr 1 15:30:35 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190143 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 15:30:35 oss2 kernel: req@ffff8104bb982400 x1392830283190143/t0 o400->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 192/384 e 0 to 1 dl 1333315835 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 15:30:35 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Apr 1 15:30:35 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.5@o2ib was lost; in progress operations using this service will fail. Apr 1 15:30:38 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client c78ae449-ef78-cf1f-38c0-3b08f0a03a36 (at 192.168.20.2@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Apr 1 15:30:38 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 15:30:42 oss2 kernel: Lustre: 12754:0:(import.c:855:ptlrpc_connect_interpret()) MGS@MGC192.168.64.5@o2ib_1 changed server handle from 0x79889ef34bf5ef20 to 0x6285a0cc05460fdc Apr 1 15:30:42 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 15:30:42 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 15:30:44 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c78ae449-ef78-cf1f-38c0-3b08f0a03a36 (at 192.168.20.2@o2ib) in 240 seconds. I think it's dead, and I am evicting it. Apr 1 15:30:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Apr 1 15:30:44 oss2 kernel: LustreError: Skipped 13 previous similar messages Apr 1 15:31:19 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 15:31:19 oss2 kernel: LustreError: Skipped 17 previous similar messages Apr 1 15:31:19 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810483fee000 x1397272694004917/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333315979 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 15:31:19 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 30 previous similar messages Apr 1 15:36:40 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 15:36:40 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 1 15:36:40 oss2 kernel: Lustre: 13233:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0009: deleting orphan objects from 234 to 257, orphan objids won't be reused any more. Apr 1 15:36:40 oss2 kernel: Lustre: 13233:0:(filter.c:3126:filter_destroy_precreated()) Skipped 5 previous similar messages Apr 1 17:06:10 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:06:38 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:06:39 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:06:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190393 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 7s ago has failed due to network error (17s prior to deadline). Apr 1 17:06:40 oss2 kernel: req@ffff81060b1aa000 x1392830283190393/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333321610 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:06:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 17:06:40 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:06:40 oss2 kernel: Lustre: 13182:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: lustre-mdtlov_UUID reconnecting Apr 1 17:06:40 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 17:06:40 oss2 kernel: LustreError: Skipped 5 previous similar messages Apr 1 17:06:40 oss2 kernel: LustreError: 13224:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810ab3bd4000 x1398082925239034/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333321700 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 17:06:40 oss2 kernel: LustreError: 13224:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Apr 1 17:06:40 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:06:40 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 17:06:46 oss2 kernel: Lustre: 13215:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: lustre-mdtlov_UUID reconnecting Apr 1 17:06:46 oss2 kernel: Lustre: 13215:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 13 previous similar messages Apr 1 17:06:46 oss2 kernel: Lustre: lustre-OST0005: received MDS connection from 192.168.64.6@o2ib Apr 1 17:06:46 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 1 17:07:22 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:07:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190414 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 17:07:40 oss2 kernel: req@ffff81060ed14800 x1392830283190414/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333321660 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:07:40 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:07:45 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:07:53 oss2 kernel: Lustre: 13291:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: lustre-mdtlov_UUID reconnecting Apr 1 17:07:53 oss2 kernel: Lustre: 13291:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 13 previous similar messages Apr 1 17:07:53 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 17:07:53 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 1 17:07:54 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:07:54 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 17:08:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 17:08:02 oss2 kernel: LustreError: Skipped 20 previous similar messages Apr 1 17:08:02 oss2 kernel: LustreError: 13256:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106954e5800 x1397268323036023/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333321782 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 17:08:02 oss2 kernel: LustreError: 13256:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20 previous similar messages Apr 1 17:08:16 oss2 kernel: Lustre: 13171:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 17:08:16 oss2 kernel: Lustre: 13171:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Apr 1 17:22:48 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:23:05 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190471 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 17:23:05 oss2 kernel: req@ffff81048982a000 x1392830283190471/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333322585 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:23:05 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 17:23:05 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:23:11 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190472 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 17:23:11 oss2 kernel: req@ffff81060ec69c00 x1392830283190472/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333322591 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:23:18 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190473 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 17:23:18 oss2 kernel: req@ffff810478fe9400 x1392830283190473/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333322598 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:23:27 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190475 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 1s ago has failed due to network error (6s prior to deadline). Apr 1 17:23:27 oss2 kernel: req@ffff8104c7552c00 x1392830283190475/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333322612 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:23:27 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 17:23:47 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190478 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 0s ago has failed due to network error (6s prior to deadline). Apr 1 17:23:47 oss2 kernel: req@ffff81047d26d400 x1392830283190478/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333322633 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:23:47 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 17:23:56 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 17:23:56 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 17:23:56 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 17:23:56 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 17:23:56 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 17:23:57 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 17:23:57 oss2 kernel: LustreError: 32247:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101bbede400 x1392830283190483/t0 o101->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:23:57 oss2 kernel: Lustre: 32524:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0018, 4 recoverable clients, 0 delayed clients, last_rcvd 30064771072 Apr 1 17:23:57 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 17:23:57 oss2 kernel: Lustre: lustre-OST0018: Now serving lustre-OST0018 on /dev/mapper/ost_lustre_24 with recovery enabled Apr 1 17:23:57 oss2 kernel: Lustre: lustre-OST0018: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:23:57 oss2 kernel: Lustre: 32524:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0018: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:23:57 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 17:23:57 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 17:23:57 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 17:23:57 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 17:23:58 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:58 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:58 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:58 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 17:23:58 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 17:23:58 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 17:23:59 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 17:23:59 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 17:23:59 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:23:59 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 17:23:59 oss2 kernel: LustreError: 465:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810552349c00 x1392830283190495/t0 o101->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:23:59 oss2 kernel: LustreError: 465:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Apr 1 17:23:59 oss2 kernel: Lustre: 688:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0010, 4 recoverable clients, 0 delayed clients, last_rcvd 30064771072 Apr 1 17:23:59 oss2 kernel: Lustre: 688:0:(filter.c:1001:filter_init_server_data()) Skipped 3 previous similar messages Apr 1 17:23:59 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 17:23:59 oss2 kernel: Lustre: lustre-OST0010: Now serving lustre-OST0010 on /dev/mapper/ost_lustre_16 with recovery enabled Apr 1 17:23:59 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:23:59 oss2 kernel: Lustre: lustre-OST0010: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:23:59 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:23:59 oss2 kernel: Lustre: 688:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0010: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:23:59 oss2 kernel: Lustre: 688:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 3 previous similar messages Apr 1 17:23:59 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 17:23:59 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 17:24:06 oss2 kernel: LustreError: 32517:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff810477809c00 x1392830283190502/t0 o501->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 264/432 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:24:06 oss2 kernel: LustreError: 32517:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Apr 1 17:24:06 oss2 kernel: Lustre: 809:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST000c, 4 recoverable clients, 0 delayed clients, last_rcvd 30064771072 Apr 1 17:24:06 oss2 kernel: Lustre: 809:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Apr 1 17:24:06 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 17:24:06 oss2 kernel: Lustre: lustre-OST000c: Now serving lustre-OST000c on /dev/mapper/ost_lustre_12 with recovery enabled Apr 1 17:24:06 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:24:06 oss2 kernel: Lustre: lustre-OST000c: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:24:06 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:24:06 oss2 kernel: Lustre: 809:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST000c: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:24:06 oss2 kernel: Lustre: 809:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Apr 1 17:24:18 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:24:19 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:24:20 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190527 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 0s ago has failed due to network error (6s prior to deadline). Apr 1 17:24:20 oss2 kernel: req@ffff8104e0c69000 x1392830283190527/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333322666 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:24:20 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Apr 1 17:24:21 oss2 kernel: Lustre: 5332:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.6@o2ib Apr 1 17:24:21 oss2 kernel: Lustre: 5332:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 1878 previous similar messages Apr 1 17:24:25 oss2 kernel: Lustre: 13227:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST000c: 3 recoverable clients remain Apr 1 17:24:27 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:24:27 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:24:31 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0008: 2 recoverable clients remain Apr 1 17:24:31 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 8 previous similar messages Apr 1 17:24:34 oss2 kernel: Lustre: 13268:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0000: 1 recoverable clients remain Apr 1 17:24:34 oss2 kernel: Lustre: 13268:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 3 previous similar messages Apr 1 17:24:37 oss2 kernel: Lustre: 13284:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 17:24:37 oss2 kernel: Lustre: 13284:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 17:24:42 oss2 kernel: Lustre: 13273:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: lustre-mdtlov_UUID reconnecting Apr 1 17:24:42 oss2 kernel: Lustre: 13273:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9 previous similar messages Apr 1 17:24:42 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 17:24:42 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 1 17:24:55 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:25:17 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:25:17 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:25:17 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:25:21 oss2 kernel: Lustre: 13226:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0014: lustre-mdtlov_UUID reconnecting Apr 1 17:25:21 oss2 kernel: Lustre: 13226:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 17:25:21 oss2 kernel: Lustre: 13226:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0014: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff810604418e00; still busy with 1 active RPCs Apr 1 17:25:21 oss2 kernel: LustreError: 13226:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106bcd9e000 x1398082925240709/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322821 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:25:21 oss2 kernel: LustreError: 13226:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 17:25:21 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:25:21 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 17:25:29 oss2 kernel: Lustre: 13176:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0000: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff8105e898ce00; still busy with 1 active RPCs Apr 1 17:25:29 oss2 kernel: Lustre: 13176:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 5 previous similar messages Apr 1 17:25:29 oss2 kernel: LustreError: 13176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106bcd9e000 x1398082925240735/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322829 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:25:29 oss2 kernel: LustreError: 13176:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 17:25:40 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000c: lustre-mdtlov_UUID reconnecting Apr 1 17:25:40 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 11 previous similar messages Apr 1 17:25:40 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000c: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff810602616200; still busy with 1 active RPCs Apr 1 17:25:40 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 5 previous similar messages Apr 1 17:25:40 oss2 kernel: LustreError: 13212:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8104f399bc00 x1398082925240743/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322840 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:25:40 oss2 kernel: LustreError: 13212:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Apr 1 17:25:51 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0000: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff8105e898ce00; still busy with 1 active RPCs Apr 1 17:25:51 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 8 previous similar messages Apr 1 17:25:51 oss2 kernel: LustreError: 13231:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3cd9c800 x1398082925240770/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322851 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:25:51 oss2 kernel: LustreError: 13231:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Apr 1 17:26:02 oss2 kernel: Lustre: 13229:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0000: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff8105e898ce00; still busy with 1 active RPCs Apr 1 17:26:02 oss2 kernel: Lustre: 13229:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 4 previous similar messages Apr 1 17:26:13 oss2 kernel: Lustre: 13294:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000c: lustre-mdtlov_UUID reconnecting Apr 1 17:26:13 oss2 kernel: Lustre: 13291:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0000: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff8105e898ce00; still busy with 1 active RPCs Apr 1 17:26:13 oss2 kernel: Lustre: 13184:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0014: refuse reconnection from lustre-mdtlov_UUID@192.168.64.6@o2ib to 0xffff810604418e00; still busy with 1 active RPCs Apr 1 17:26:13 oss2 kernel: LustreError: 13265:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3d681800 x1398082925240788/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322873 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:26:13 oss2 kernel: Lustre: 13184:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 4 previous similar messages Apr 1 17:26:13 oss2 kernel: LustreError: 13265:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9 previous similar messages Apr 1 17:26:13 oss2 kernel: Lustre: 13294:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 24 previous similar messages Apr 1 17:26:13 oss2 kernel: LustreError: 13199:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810613cce850 x1398082925240789/t0 o8->lustre-mdtlov_UUID@MGC192.168.64.5@o2ib_1:0/0 lens 368/264 e 0 to 0 dl 1333322873 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:26:13 oss2 kernel: LustreError: 13199:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 1 17:26:13 oss2 kernel: Lustre: 13291:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message Apr 1 17:26:15 oss2 kernel: Lustre: lustre-OST0014: Recovery period over after 1:50, of 4 clients 4 recovered and 0 were evicted. Apr 1 17:26:15 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:26:15 oss2 kernel: Lustre: lustre-OST0014: sending delayed replies to recovered clients Apr 1 17:26:15 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 1 17:26:19 oss2 kernel: Lustre: Failing over lustre-OST0018 Apr 1 17:26:19 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:26:21 oss2 kernel: Lustre: lustre-OST0018: shutting down for failover; client state will be preserved. Apr 1 17:26:21 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:26:21 oss2 kernel: Lustre: OST lustre-OST0018 has stopped. Apr 1 17:26:21 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:26:21 oss2 kernel: Lustre: server umount lustre-OST0018 complete Apr 1 17:26:21 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:26:21 oss2 kernel: Lustre: Failing over lustre-OST0008 Apr 1 17:26:21 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:26:23 oss2 kernel: Lustre: lustre-OST0008: shutting down for failover; client state will be preserved. Apr 1 17:26:23 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:26:23 oss2 kernel: Lustre: OST lustre-OST0008 has stopped. Apr 1 17:26:23 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:26:23 oss2 kernel: Lustre: server umount lustre-OST0008 complete Apr 1 17:26:23 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:26:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Apr 1 17:26:24 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 17:26:34 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 17:26:34 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 17:26:40 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 17:26:40 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 17:26:56 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Apr 1 17:26:56 oss2 kernel: LustreError: Skipped 20 previous similar messages Apr 1 17:38:43 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:39:08 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:39:09 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:39:09 oss2 kernel: Lustre: 13192:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 9d98c294-b076-0f80-e073-d2e0279f8b2e reconnecting Apr 1 17:39:09 oss2 kernel: Lustre: 13192:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 17:39:09 oss2 kernel: Lustre: 13264:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 9d98c294-b076-0f80-e073-d2e0279f8b2e@192.168.2.2@o2ib to 0xffff8107fdca1600; still busy with 1 active RPCs Apr 1 17:39:09 oss2 kernel: LustreError: 13264:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b04c19400 x1397284104409081/t0 o8->9d98c294-b076-0f80-e073-d2e0279f8b2e@NET_0x50000c0a80202_UUID:0/0 lens 368/264 e 0 to 0 dl 1333323649 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 17:39:09 oss2 kernel: LustreError: 13264:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 63 previous similar messages Apr 1 17:39:10 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190603 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 7s ago has failed due to network error (17s prior to deadline). Apr 1 17:39:10 oss2 kernel: req@ffff8101dcf35000 x1392830283190603/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333323560 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:39:10 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message Apr 1 17:39:10 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:39:12 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.6@o2ib Apr 1 17:39:12 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Apr 1 17:39:12 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 17:39:12 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:39:12 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 17:39:16 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 17:39:16 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 1 17:39:53 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:40:12 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190624 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 19s ago has timed out (19s prior to deadline). Apr 1 17:40:12 oss2 kernel: req@ffff8106072f1800 x1392830283190624/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333323612 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:40:12 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:40:18 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:40:20 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:40:22 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:40:25 oss2 kernel: Lustre: 13271:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 1 17:40:25 oss2 kernel: Lustre: 13208:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0011: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 1 17:40:25 oss2 kernel: Lustre: 13208:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 22 previous similar messages Apr 1 17:40:25 oss2 kernel: Lustre: 13271:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 22 previous similar messages Apr 1 17:40:25 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 17:40:25 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 1 17:40:28 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:40:28 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 17:40:32 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 17:40:32 oss2 kernel: LustreError: Skipped 13 previous similar messages Apr 1 17:40:32 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c9e850 x1397268323038837/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333323732 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 17:40:32 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 14 previous similar messages Apr 1 17:40:43 oss2 kernel: Lustre: 13226:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 1 17:40:43 oss2 kernel: Lustre: 13226:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Apr 1 17:56:54 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:57:15 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190685 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 17:57:15 oss2 kernel: req@ffff81063e152800 x1392830283190685/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333324635 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:57:15 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 17:57:15 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 17:57:21 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190686 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 17:57:21 oss2 kernel: req@ffff810623b72800 x1392830283190686/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333324641 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:57:28 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190687 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 17:57:28 oss2 kernel: req@ffff8106208e6000 x1392830283190687/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333324648 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 17:57:43 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190690 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 0s ago has failed due to network error (6s prior to deadline). Apr 1 17:57:43 oss2 kernel: req@ffff81061932f400 x1392830283190690/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333324669 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:57:43 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 17:58:04 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190693 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 0s ago has failed due to network error (6s prior to deadline). Apr 1 17:58:04 oss2 kernel: req@ffff81061e40d400 x1392830283190693/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333324690 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 17:58:04 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 17:58:07 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 17:58:07 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 17:58:07 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 17:58:07 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:07 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: LustreError: 13686:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8103bdc61000 x1392830283190697/t0 o101->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:58:08 oss2 kernel: LustreError: 13686:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 23 previous similar messages Apr 1 17:58:08 oss2 kernel: Lustre: 14028:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0018, 4 recoverable clients, 0 delayed clients, last_rcvd 38654705664 Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: Lustre: lustre-OST0018: Now serving lustre-OST0018 on /dev/mapper/ost_lustre_24 with recovery enabled Apr 1 17:58:08 oss2 kernel: Lustre: lustre-OST0018: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:58:08 oss2 kernel: Lustre: 14028:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0018: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:58:08 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:08 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Apr 1 17:58:09 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Apr 1 17:58:09 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:58:09 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:58:09 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Apr 1 17:58:09 oss2 kernel: LustreError: 14273:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8101f755bc00 x1392830283190709/t0 o101->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:58:09 oss2 kernel: LustreError: 14273:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Apr 1 17:58:09 oss2 kernel: Lustre: 14536:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0010, 4 recoverable clients, 0 delayed clients, last_rcvd 38654705664 Apr 1 17:58:09 oss2 kernel: Lustre: 14536:0:(filter.c:1001:filter_init_server_data()) Skipped 3 previous similar messages Apr 1 17:58:09 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Apr 1 17:58:09 oss2 kernel: Lustre: lustre-OST0010: Now serving lustre-OST0010 on /dev/mapper/ost_lustre_16 with recovery enabled Apr 1 17:58:09 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:58:09 oss2 kernel: Lustre: lustre-OST0010: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:58:09 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:58:09 oss2 kernel: Lustre: 14536:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0010: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:58:09 oss2 kernel: Lustre: 14536:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 3 previous similar messages Apr 1 17:58:09 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Apr 1 17:58:09 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Apr 1 17:58:11 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 17:58:12 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 17:58:14 oss2 kernel: Lustre: 13290:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 17:58:14 oss2 kernel: Lustre: 13290:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8 previous similar messages Apr 1 17:58:14 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 17:58:14 oss2 kernel: LustreError: Skipped 7 previous similar messages Apr 1 17:58:14 oss2 kernel: LustreError: 13200:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81051093b800 x1397268323040106/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333324794 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 17:58:14 oss2 kernel: LustreError: 13200:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Apr 1 17:58:15 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Conn race 192.168.64.6@o2ib Apr 1 17:58:15 oss2 kernel: Lustre: 5314:0:(o2iblnd_cb.c:2257:kiblnd_passive_connect()) Skipped 1 previous similar message Apr 1 17:58:20 oss2 kernel: Lustre: 13208:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0000: lustre-mdtlov_UUID reconnecting Apr 1 17:58:20 oss2 kernel: Lustre: 13208:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 14 previous similar messages Apr 1 17:58:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 17:58:20 oss2 kernel: LustreError: Skipped 3 previous similar messages Apr 1 17:58:20 oss2 kernel: LustreError: 13217:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b2b289800 x1398082925243555/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333324800 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 17:58:20 oss2 kernel: LustreError: 13217:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Apr 1 17:58:20 oss2 kernel: Lustre: 13207:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0000: 3 recoverable clients remain Apr 1 17:58:20 oss2 kernel: Lustre: 13207:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 5 previous similar messages Apr 1 17:58:22 oss2 kernel: LustreError: 14268:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81046b7df800 x1392830283190716/t0 o101->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Apr 1 17:58:22 oss2 kernel: LustreError: 14268:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Apr 1 17:58:22 oss2 kernel: Lustre: 14792:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0008, 4 recoverable clients, 0 delayed clients, last_rcvd 38654705664 Apr 1 17:58:22 oss2 kernel: Lustre: 14792:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Apr 1 17:58:22 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Apr 1 17:58:22 oss2 kernel: Lustre: lustre-OST0008: Now serving lustre-OST0008 on /dev/mapper/ost_lustre_8 with recovery enabled Apr 1 17:58:22 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:58:22 oss2 kernel: Lustre: lustre-OST0008: Will be in recovery for at least 5:00, or until 4 clients reconnect Apr 1 17:58:22 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 17:58:22 oss2 kernel: Lustre: 14792:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0008: quotaon failed because quota files don't exist, please run quotacheck firstly Apr 1 17:58:22 oss2 kernel: Lustre: 14792:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Apr 1 17:58:23 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 17:58:23 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 17:58:23 oss2 kernel: Lustre: 13172:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0014: 1 recoverable clients remain Apr 1 17:58:23 oss2 kernel: Lustre: 13172:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 10 previous similar messages Apr 1 17:58:30 oss2 kernel: Lustre: 13231:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: lustre-mdtlov_UUID reconnecting Apr 1 17:58:30 oss2 kernel: Lustre: 13231:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 22 previous similar messages Apr 1 17:58:30 oss2 kernel: Lustre: lustre-OST0005: received MDS connection from 192.168.64.6@o2ib Apr 1 17:58:30 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 1 17:58:35 oss2 kernel: Lustre: 13286:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0008: 2 recoverable clients remain Apr 1 17:58:35 oss2 kernel: Lustre: 13286:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 4 previous similar messages Apr 1 17:58:35 oss2 kernel: Lustre: lustre-OST0010: Recovery period over after 0:20, of 4 clients 4 recovered and 0 were evicted. Apr 1 17:58:35 oss2 kernel: Lustre: Skipped 6 previous similar messages Apr 1 17:58:35 oss2 kernel: Lustre: lustre-OST0000: sending delayed replies to recovered clients Apr 1 17:58:35 oss2 kernel: Lustre: lustre-OST0004: sending delayed replies to recovered clients Apr 1 17:58:35 oss2 kernel: Lustre: Skipped 7 previous similar messages Apr 1 17:58:35 oss2 kernel: Lustre: Skipped 7 previous similar messages Apr 1 17:58:35 oss2 kernel: Lustre: lustre-OST0010: received MDS connection from 192.168.64.6@o2ib Apr 1 17:58:35 oss2 kernel: Lustre: Skipped 6 previous similar messages Apr 1 17:58:41 oss2 kernel: Lustre: 13236:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0008: 1 recoverable clients remain Apr 1 17:58:41 oss2 kernel: Lustre: lustre-OST0008: Recovery period over after 0:18, of 4 clients 4 recovered and 0 were evicted. Apr 1 17:58:41 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:58:41 oss2 kernel: Lustre: lustre-OST0008: sending delayed replies to recovered clients Apr 1 17:58:41 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:58:41 oss2 kernel: Lustre: lustre-OST0008: received MDS connection from 192.168.64.6@o2ib Apr 1 17:58:41 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 17:58:51 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 17:59:14 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 18:00:27 oss2 kernel: Lustre: Failing over lustre-OST0000 Apr 1 18:00:27 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 18:00:28 oss2 kernel: LustreError: 13202:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810613c4b050 x1397284104410764/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1333324847 ref 1 fl Interpret:H/0/0 rc -107/0 Apr 1 18:00:28 oss2 kernel: LustreError: 13202:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 1 18:00:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) Apr 1 18:00:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (stopping) Apr 1 18:00:28 oss2 kernel: LustreError: Skipped 1 previous similar message Apr 1 18:00:28 oss2 kernel: LustreError: Skipped 1 previous similar message Apr 1 18:00:29 oss2 kernel: Lustre: lustre-OST0018: shutting down for failover; client state will be preserved. Apr 1 18:00:29 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 18:00:29 oss2 kernel: Lustre: OST lustre-OST0018 has stopped. Apr 1 18:00:29 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 18:00:29 oss2 kernel: Lustre: server umount lustre-OST0018 complete Apr 1 18:00:29 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 18:00:29 oss2 kernel: Lustre: Failing over lustre-OST000c Apr 1 18:00:29 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 18:00:31 oss2 kernel: Lustre: lustre-OST0010: shutting down for failover; client state will be preserved. Apr 1 18:00:31 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 18:00:31 oss2 kernel: Lustre: OST lustre-OST0010 has stopped. Apr 1 18:00:31 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 18:00:31 oss2 kernel: Lustre: server umount lustre-OST0010 complete Apr 1 18:00:31 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 18:00:44 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b146ae800 x1397268323040567/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1333324870 ref 1 fl Interpret:H/0/0 rc -107/0 Apr 1 18:00:44 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10 previous similar messages Apr 1 18:00:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 18:00:44 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 1 18:00:53 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 1 18:00:53 oss2 kernel: LustreError: Skipped 24 previous similar messages Apr 1 18:12:20 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 18:12:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190814 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 18:12:40 oss2 kernel: req@ffff810416c3b800 x1392830283190814/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333325560 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 18:12:40 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Apr 1 18:12:40 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 18:12:46 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190815 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 18:12:46 oss2 kernel: req@ffff81061ab12000 x1392830283190815/t0 o250->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 368/584 e 0 to 1 dl 1333325566 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 18:12:48 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 18:12:49 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 18:12:50 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 1 18:12:50 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 23 previous similar messages Apr 1 18:12:50 oss2 kernel: Lustre: 13260:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from bd098490-ea0c-c4d1-0417-0e62318f0421@192.168.2.4@o2ib to 0xffff810c379d7c00; still busy with 1 active RPCs Apr 1 18:12:50 oss2 kernel: LustreError: 13260:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81061f39b000 x1397272694017001/t0 o8->bd098490-ea0c-c4d1-0417-0e62318f0421@NET_0x50000c0a80204_UUID:0/0 lens 368/264 e 0 to 0 dl 1333325670 ref 1 fl Interpret:/0/0 rc -16/0 Apr 1 18:12:50 oss2 kernel: LustreError: 13260:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 46 previous similar messages Apr 1 18:12:53 oss2 kernel: Lustre: 13170:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 9d98c294-b076-0f80-e073-d2e0279f8b2e reconnecting Apr 1 18:12:53 oss2 kernel: Lustre: 13170:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 18:12:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Apr 1 18:12:54 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 1 18:12:54 oss2 kernel: LustreError: 13263:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c41c50 x1398082925244764/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333325674 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 18:12:54 oss2 kernel: LustreError: 13263:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 1 18:13:01 oss2 kernel: Lustre: 13192:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: lustre-mdtlov_UUID reconnecting Apr 1 18:13:01 oss2 kernel: Lustre: 13192:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 17 previous similar messages Apr 1 18:13:01 oss2 kernel: Lustre: lustre-OST0005: received MDS connection from 192.168.64.6@o2ib Apr 1 18:13:01 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 18:13:01 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 1 18:13:07 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283190818 sent from MGC192.168.64.5@o2ib to NID 192.168.64.5@o2ib 6s ago has timed out (6s prior to deadline). Apr 1 18:13:07 oss2 kernel: req@ffff810615716800 x1392830283190818/t0 o250->MGS@MGC192.168.64.5@o2ib_0:26/25 lens 368/584 e 0 to 1 dl 1333325587 ref 1 fl Rpc:N/0/0 rc 0/0 Apr 1 18:13:07 oss2 kernel: Lustre: 12754:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Apr 1 18:13:08 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 18:13:08 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 18:13:31 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 18:13:55 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283195521 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 18:13:55 oss2 kernel: req@ffff810619057800 x1392830283195521/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333325635 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 18:13:55 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 18:13:58 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 18:13:59 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 18:14:00 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 18:14:00 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Apr 1 18:14:04 oss2 kernel: Lustre: lustre-OST0005: received MDS connection from 192.168.64.6@o2ib Apr 1 18:14:04 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 1 18:14:09 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 18:14:09 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 18:18:33 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 18:18:55 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283195554 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 18:18:55 oss2 kernel: req@ffff810618c96000 x1392830283195554/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333325935 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 18:18:55 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Apr 1 18:18:55 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 18:19:08 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 18:19:09 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 18:19:12 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 1 18:19:12 oss2 kernel: Lustre: 13281:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 11 previous similar messages Apr 1 18:19:20 oss2 kernel: Lustre: lustre-OST000d: received MDS connection from 192.168.64.6@o2ib Apr 1 18:19:20 oss2 kernel: Lustre: Skipped 3 previous similar messages Apr 1 18:19:23 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 18:19:23 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 18:19:29 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 1 18:19:29 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 18:19:29 oss2 kernel: LustreError: 13236:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810311945000 x1398082925245436/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333326069 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 18:19:29 oss2 kernel: LustreError: 13236:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 18:19:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Apr 1 18:19:48 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 18:19:48 oss2 kernel: LustreError: 13260:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810616b04800 x1398082925245476/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333326088 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 18:19:48 oss2 kernel: LustreError: 13260:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 18:19:52 oss2 kernel: ib_srp: ASYNC event= 10 on device= mlx4_0 Apr 1 18:20:10 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1392830283195579 sent from MGC192.168.64.5@o2ib to NID 192.168.64.6@o2ib 17s ago has timed out (17s prior to deadline). Apr 1 18:20:10 oss2 kernel: req@ffff810625b08000 x1392830283195579/t0 o400->MGS@MGC192.168.64.5@o2ib_1:26/25 lens 192/384 e 0 to 1 dl 1333326010 ref 2 fl Rpc:N/0/0 rc 0/0 Apr 1 18:20:10 oss2 kernel: Lustre: 12753:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Apr 1 18:20:10 oss2 kernel: LustreError: 166-1: MGC192.168.64.5@o2ib: Connection to service MGS via nid 192.168.64.6@o2ib was lost; in progress operations using this service will fail. Apr 1 18:20:18 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 1 18:20:19 oss2 kernel: ib_srp: ASYNC event= 9 on device= mlx4_0 Apr 1 18:20:20 oss2 kernel: Lustre: 13279:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: lustre-mdtlov_UUID reconnecting Apr 1 18:20:20 oss2 kernel: Lustre: 13279:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 14 previous similar messages Apr 1 18:20:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Apr 1 18:20:20 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 18:20:20 oss2 kernel: LustreError: 13213:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b457a8400 x1398082925245509/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333326120 ref 1 fl Interpret:/0/0 rc -19/0 Apr 1 18:20:20 oss2 kernel: LustreError: 13213:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 1 18:20:20 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Apr 1 18:20:20 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 1 18:20:24 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Reactivating import Apr 1 18:20:24 oss2 kernel: Lustre: MGC192.168.64.5@o2ib: Connection restored to service MGS using nid 192.168.64.6@o2ib. Apr 1 18:20:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Apr 1 18:20:38 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 1 18:20:43 oss2 kernel: Lustre: lustre-OST0011: received MDS connection from 192.168.64.6@o2ib Apr 1 18:20:43 oss2 kernel: Lustre: Skipped 1 previous similar message Apr 2 06:33:22 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client ec7420d9-5818-4810-22e7-5944d82f2eec (at 192.168.11.20@o2ib) in 230 seconds. I think it's dead, and I am evicting it. Apr 2 06:33:22 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 2 07:37:55 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 519bf802-6215-8d93-a93e-7a9ccc8acabe (at 192.168.11.20@o2ib) in 233 seconds. I think it's dead, and I am evicting it. Apr 2 07:37:55 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 2 09:29:09 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client ac446037-bed4-73ac-dc91-4c21953c50d0 (at 192.168.11.20@o2ib) in 238 seconds. I think it's dead, and I am evicting it. Apr 2 09:29:09 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 2 10:11:39 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client e761cc4d-18a3-b7b1-4972-3e9953f7f05b (at 192.168.11.20@o2ib) in 246 seconds. I think it's dead, and I am evicting it. Apr 2 10:11:39 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 2 10:55:00 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 0576f498-f3ac-5f5d-20f9-5dd8f5815da5 (at 192.168.11.20@o2ib) in 231 seconds. I think it's dead, and I am evicting it. Apr 2 10:55:00 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 2 12:31:15 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 79223c5e-582e-a1fa-e65a-bdfd152717aa (at 192.168.11.20@o2ib) in 230 seconds. I think it's dead, and I am evicting it. Apr 2 12:31:15 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 2 14:51:53 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Apr 2 15:17:55 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client ee42681f-e0aa-7725-2a67-80df7a8fc3af (at 192.168.11.20@o2ib) in 233 seconds. I think it's dead, and I am evicting it. Apr 2 15:17:55 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 2 15:34:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Apr 2 15:34:38 oss2 kernel: LustreError: Skipped 4 previous similar messages Apr 2 15:34:38 oss2 kernel: LustreError: 13251:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81088f35b000 x1397268323125313/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333402578 ref 1 fl Interpret:/0/0 rc -19/0 Apr 2 15:34:38 oss2 kernel: LustreError: 13251:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 11 previous similar messages Apr 2 15:40:54 oss2 kernel: Lustre: 13256:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: fdfe6d24-762a-2357-5e80-1f62efef7d0b reconnecting Apr 2 15:40:54 oss2 kernel: Lustre: 13256:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 19 previous similar messages Apr 2 15:45:00 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client fdfe6d24-762a-2357-5e80-1f62efef7d0b (at 192.168.11.18@o2ib) in 231 seconds. I think it's dead, and I am evicting it. Apr 2 15:45:00 oss2 kernel: Lustre: Skipped 11 previous similar messages Apr 2 16:07:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Apr 2 16:07:31 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 2 16:07:31 oss2 kernel: LustreError: 13269:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3d867000 x1397268323127854/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333404551 ref 1 fl Interpret:/0/0 rc -19/0 Apr 2 16:07:31 oss2 kernel: LustreError: 13269:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 2 17:55:09 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: bd098490-ea0c-c4d1-0417-0e62318f0421 reconnecting Apr 2 17:55:09 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Apr 3 17:22:03 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 7b0115f3-c707-3d24-077b-46fdb9c9e63d (at 192.168.11.20@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Apr 3 17:22:03 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 3 17:22:03 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 7b0115f3-c707-3d24-077b-46fdb9c9e63d (at 192.168.11.20@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Apr 9 07:41:10 oss2 kernel: Lustre: 13263:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:41:10 oss2 kernel: Lustre: 13263:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Apr 9 07:41:34 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:41:34 oss2 kernel: LustreError: Skipped 6 previous similar messages Apr 9 07:41:34 oss2 kernel: LustreError: 13240:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061a3fe800 x1398349650907498/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333978994 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:41:34 oss2 kernel: LustreError: 13240:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Apr 9 07:41:59 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:42:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:42:10 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 9 07:42:10 oss2 kernel: LustreError: 13293:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061175cc50 x1398349650907535/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979030 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:42:10 oss2 kernel: LustreError: 13293:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Apr 9 07:42:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Apr 9 07:42:37 oss2 kernel: LustreError: 13289:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109e28ac400 x1398349650907557/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979057 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:43:36 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:43:36 oss2 kernel: LustreError: Skipped 1 previous similar message Apr 9 07:43:36 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810623b72800 x1398349650907617/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979116 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:43:36 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 9 07:43:50 oss2 kernel: Lustre: 13279:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:45:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Apr 9 07:45:09 oss2 kernel: LustreError: 13206:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a10451c00 x1398349650907736/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979209 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:45:14 oss2 kernel: Lustre: 13252:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:45:14 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Apr 9 07:45:14 oss2 kernel: LustreError: 13198:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109aa5a3000 x1398349650907740/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979214 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:45:34 oss2 kernel: Lustre: 13272:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:45:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:45:38 oss2 kernel: LustreError: Skipped 1 previous similar message Apr 9 07:45:38 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105b8653800 x1398349650907809/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979238 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:45:38 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Apr 9 07:45:46 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:45:46 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Apr 9 07:46:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:46:08 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 9 07:46:08 oss2 kernel: LustreError: 13233:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106157a3400 x1398349650907859/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979268 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:46:08 oss2 kernel: LustreError: 13233:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Apr 9 07:46:55 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:46:55 oss2 kernel: LustreError: Skipped 2 previous similar messages Apr 9 07:46:55 oss2 kernel: LustreError: 13195:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109cf7fa400 x1398349650907906/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979315 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:46:55 oss2 kernel: LustreError: 13195:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Apr 9 07:47:01 oss2 kernel: Lustre: 13263:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:47:40 oss2 kernel: Lustre: 13184:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:54:56 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Apr 9 07:54:56 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613cdcc50 x1398349650908524/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1333979796 ref 1 fl Interpret:/0/0 rc -19/0 Apr 9 07:56:27 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 21ff134f-4258-e4fe-cd92-2c91f6f0725d (at 192.168.11.19@o2ib) in 233 seconds. I think it's dead, and I am evicting it. Apr 9 07:56:27 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 9 07:56:46 oss2 kernel: Lustre: 13224:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 21ff134f-4258-e4fe-cd92-2c91f6f0725d reconnecting Apr 9 07:57:20 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 21ff134f-4258-e4fe-cd92-2c91f6f0725d (at 192.168.11.19@o2ib) in 229 seconds. I think it's dead, and I am evicting it. Apr 9 07:57:28 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 21ff134f-4258-e4fe-cd92-2c91f6f0725d (at 192.168.11.19@o2ib) in 230 seconds. I think it's dead, and I am evicting it. Apr 9 07:57:28 oss2 kernel: Lustre: Skipped 2 previous similar messages Apr 20 11:46:52 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 21ff134f-4258-e4fe-cd92-2c91f6f0725d (at 192.168.11.19@o2ib) in 234 seconds. I think it's dead, and I am evicting it. Apr 20 14:48:49 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client d10ebaef-45d3-0427-de3a-452f804d7b12 (at 192.168.11.18@o2ib) in 229 seconds. I think it's dead, and I am evicting it. Apr 20 14:48:49 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 20 14:48:49 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client d10ebaef-45d3-0427-de3a-452f804d7b12 (at 192.168.11.18@o2ib) in 229 seconds. I think it's dead, and I am evicting it. Apr 23 14:17:27 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client de0094b3-755c-35c1-72e5-6e9082062a2f (at 192.168.11.19@o2ib) in 230 seconds. I think it's dead, and I am evicting it. Apr 23 14:17:27 oss2 kernel: Lustre: Skipped 4 previous similar messages Apr 23 14:24:35 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 24cb0db4-cae2-1b17-b18b-6f51486923e6 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Apr 23 14:24:35 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 24 12:05:14 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client cf883d21-d26d-4c52-43c4-f060f4060c3a (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Apr 24 12:05:14 oss2 kernel: Lustre: Skipped 5 previous similar messages Apr 24 13:31:04 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 385d7696-482d-dbb9-6626-1d5cedd15895 (at 192.168.11.18@o2ib) in 229 seconds. I think it's dead, and I am evicting it. Apr 24 13:31:04 oss2 kernel: Lustre: Skipped 5 previous similar messages May 4 13:12:54 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c126ddf6-6860-9732-beb9-49ce18d14bb5 (at 192.168.11.2@o2ib) in 233 seconds. I think it's dead, and I am evicting it. May 4 13:12:54 oss2 kernel: Lustre: Skipped 5 previous similar messages May 4 13:12:54 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client d43f5d47-0786-beaa-1990-bb850ece5d10 (at 192.168.11.3@o2ib) in 225 seconds. I think it's dead, and I am evicting it. May 4 13:23:50 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client e6078687-4e44-2f3d-82c3-5552c09f6356 (at 192.168.2.102@o2ib) in 229 seconds. I think it's dead, and I am evicting it. May 4 13:23:50 oss2 kernel: Lustre: Skipped 10 previous similar messages May 4 13:27:18 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 4 13:27:18 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 13:27:18 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810619f7bc00 x1401076187136058/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336159738 ref 1 fl Interpret:/0/0 rc -19/0 May 4 13:27:18 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 13:27:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 13:27:43 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 13:27:43 oss2 kernel: LustreError: 13263:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611e0b050 x1401076187136081/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336159763 ref 1 fl Interpret:/0/0 rc -19/0 May 4 13:27:43 oss2 kernel: LustreError: 13263:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 13:31:02 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 13:31:02 oss2 kernel: Lustre: Skipped 4 previous similar messages May 4 13:46:05 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 13:46:05 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 13:46:05 oss2 kernel: LustreError: 13268:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81059eafc400 x1401076241663591/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336160865 ref 1 fl Interpret:/0/0 rc -19/0 May 4 13:46:05 oss2 kernel: LustreError: 13268:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 13:48:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 4 13:48:09 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 13:48:09 oss2 kernel: LustreError: 13173:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810344cf8000 x1401076241663720/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336160989 ref 1 fl Interpret:/0/0 rc -19/0 May 4 13:48:09 oss2 kernel: LustreError: 13173:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 13:49:52 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 13:49:52 oss2 kernel: Lustre: Skipped 4 previous similar messages May 4 13:54:22 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) May 4 13:54:22 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 13:54:22 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063db97400 x1401076241664112/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336161362 ref 1 fl Interpret:/0/0 rc -19/0 May 4 13:54:22 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 13:58:48 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. May 4 14:38:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 14:38:37 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 14:38:37 oss2 kernel: LustreError: 13290:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b2330b800 x1401076241666837/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336164017 ref 1 fl Interpret:/0/0 rc -19/0 May 4 14:38:37 oss2 kernel: LustreError: 13290:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 14:42:58 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. May 4 14:42:58 oss2 kernel: Lustre: Skipped 4 previous similar messages May 4 14:43:10 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 14:43:10 oss2 kernel: Lustre: Skipped 4 previous similar messages May 4 14:54:40 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 14:54:40 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 14:54:40 oss2 kernel: LustreError: 13242:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81053eff1000 x1401076241668121/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336164980 ref 1 fl Interpret:/0/0 rc -19/0 May 4 14:54:40 oss2 kernel: LustreError: 13242:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 14:54:45 oss2 kernel: Lustre: 13237:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 999a9a1c-2fe9-5ec0-2960-5ea09b204821 reconnecting May 4 14:55:28 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 14:58:32 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 14:58:32 oss2 kernel: Lustre: Skipped 3 previous similar messages May 4 14:59:51 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 4 14:59:51 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 14:59:51 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b5c761800 x1401076241668414/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336165291 ref 1 fl Interpret:/0/0 rc -19/0 May 4 14:59:51 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 15:04:12 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 231 seconds. I think it's dead, and I am evicting it. May 4 15:15:29 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 4 15:15:34 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. May 4 15:17:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 15:17:28 oss2 kernel: LustreError: Skipped 6 previous similar messages May 4 15:17:28 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b54825000 x1401076241796282/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336166348 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:17:28 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 4 15:17:53 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 4 15:17:53 oss2 kernel: LustreError: 13175:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061175cc50 x1401076241796283/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336166373 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:18:01 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 4 15:18:01 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81087bc30800 x1401076241796316/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336166381 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:21:42 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 229 seconds. I think it's dead, and I am evicting it. May 4 15:21:42 oss2 kernel: Lustre: Skipped 3 previous similar messages May 4 15:32:23 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 4 15:32:23 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810598615400 x1401076241797165/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336167243 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:32:36 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) May 4 15:32:36 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b3221f000 x1401076241797175/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336167256 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:33:26 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 4 15:33:26 oss2 kernel: LustreError: Skipped 4 previous similar messages May 4 15:33:26 oss2 kernel: LustreError: 13222:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101e6ecb400 x1401076241797229/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336167306 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:33:26 oss2 kernel: LustreError: 13222:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 4 15:33:26 oss2 kernel: Lustre: 13176:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 999a9a1c-2fe9-5ec0-2960-5ea09b204821 reconnecting May 4 15:34:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) May 4 15:34:50 oss2 kernel: LustreError: Skipped 4 previous similar messages May 4 15:34:50 oss2 kernel: LustreError: 13291:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106c5f42400 x1401076241797356/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336167390 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:34:50 oss2 kernel: LustreError: 13291:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 4 15:35:03 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 999a9a1c-2fe9-5ec0-2960-5ea09b204821 reconnecting May 4 15:37:14 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. May 4 15:37:14 oss2 kernel: Lustre: Skipped 5 previous similar messages May 4 15:39:18 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 999a9a1c-2fe9-5ec0-2960-5ea09b204821 (at 192.168.2.102@o2ib) in 229 seconds. I think it's dead, and I am evicting it. May 4 15:39:18 oss2 kernel: Lustre: Skipped 4 previous similar messages May 4 15:39:57 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) May 4 15:39:57 oss2 kernel: LustreError: Skipped 3 previous similar messages May 4 15:39:57 oss2 kernel: LustreError: 13209:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b44981400 x1401076241797703/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1336167697 ref 1 fl Interpret:/0/0 rc -19/0 May 4 15:39:57 oss2 kernel: LustreError: 13209:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages May 10 10:12:31 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 May 10 11:23:08 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 May 17 13:28:03 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 29d249d2-cdd2-d751-b896-73f8ae0c2264 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 17 13:28:03 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 29d249d2-cdd2-d751-b896-73f8ae0c2264 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 17 13:28:03 oss2 kernel: Lustre: Skipped 2 previous similar messages May 22 13:49:13 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0011: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 13:55:13 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 13:55:13 oss2 kernel: Lustre: Skipped 2 previous similar messages May 22 14:01:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 14:01:48 oss2 kernel: LustreError: 13268:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810718041400 x1400143561464331/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337717008 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:02:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 14:02:48 oss2 kernel: LustreError: Skipped 5 previous similar messages May 22 14:02:48 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c52450 x1400143561464402/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337717068 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:02:48 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages May 22 14:02:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 14:02:54 oss2 kernel: LustreError: 13192:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81057ae21000 x1400143561464405/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337717074 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:03:05 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 14:03:05 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 14:03:05 oss2 kernel: LustreError: 13202:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81044e49ec00 x1400143561464410/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337717085 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:03:05 oss2 kernel: LustreError: 13202:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 14:07:18 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 14:07:18 oss2 kernel: Lustre: Skipped 5 previous similar messages May 22 14:20:16 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) May 22 14:20:16 oss2 kernel: LustreError: Skipped 3 previous similar messages May 22 14:20:16 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b8c63d400 x1400143561465359/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337718116 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:20:16 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages May 22 14:21:07 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 14:21:07 oss2 kernel: LustreError: Skipped 4 previous similar messages May 22 14:21:07 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106cbfd3800 x1400143561465392/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337718167 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:21:07 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 22 14:21:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 22 14:21:08 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 14:21:08 oss2 kernel: LustreError: 13220:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bb329b800 x1400143561465409/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337718168 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:21:08 oss2 kernel: LustreError: 13220:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 14:24:54 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 14:24:54 oss2 kernel: Lustre: Skipped 5 previous similar messages May 22 14:24:55 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 14:24:55 oss2 kernel: Lustre: Skipped 1 previous similar message May 22 14:35:40 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 22 14:35:40 oss2 kernel: LustreError: Skipped 4 previous similar messages May 22 14:35:40 oss2 kernel: LustreError: 13225:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b7ad1fc00 x1400143561466316/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337719040 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:35:40 oss2 kernel: LustreError: 13225:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 22 14:39:10 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 14:39:10 oss2 kernel: Lustre: Skipped 3 previous similar messages May 22 14:39:27 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 14:50:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 14:50:31 oss2 kernel: LustreError: Skipped 4 previous similar messages May 22 14:50:31 oss2 kernel: LustreError: 13265:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81035b0b3c00 x1400143561467087/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337719931 ref 1 fl Interpret:/0/0 rc -19/0 May 22 14:50:31 oss2 kernel: LustreError: 13265:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 22 14:51:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 22 14:51:02 oss2 kernel: LustreError: 13176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810add588c00 x1400143561467125/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337719962 ref 1 fl Interpret:/0/0 rc -19/0 May 22 15:23:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 15:23:45 oss2 kernel: LustreError: Skipped 3 previous similar messages May 22 15:23:45 oss2 kernel: LustreError: 13281:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b32b39c00 x1400143561471053/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337721925 ref 1 fl Interpret:/0/0 rc -19/0 May 22 15:23:45 oss2 kernel: LustreError: 13281:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages May 22 15:23:46 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) May 22 15:23:46 oss2 kernel: LustreError: Skipped 4 previous similar messages May 22 15:23:46 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bccd7d800 x1400143561471073/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337721926 ref 1 fl Interpret:/0/0 rc -19/0 May 22 15:23:46 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 22 15:24:00 oss2 kernel: Lustre: 13193:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 15:24:00 oss2 kernel: Lustre: 13193:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message May 22 15:24:01 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 15:24:01 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages May 22 17:33:41 oss2 kernel: Lustre: 13223:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:33:41 oss2 kernel: Lustre: 13223:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages May 22 17:33:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 17:33:48 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 17:33:48 oss2 kernel: LustreError: 13254:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810615716800 x1400143561482256/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337729728 ref 1 fl Interpret:/0/0 rc -19/0 May 22 17:33:48 oss2 kernel: LustreError: 13254:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 17:34:01 oss2 kernel: Lustre: 13178:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:34:12 oss2 kernel: Lustre: 13289:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:34:14 oss2 kernel: Lustre: 13231:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:58:41 oss2 kernel: Lustre: 13254:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:58:41 oss2 kernel: Lustre: 13254:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages May 22 17:58:42 oss2 kernel: Lustre: 13220:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0011: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 17:58:42 oss2 kernel: Lustre: 13220:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages May 22 18:06:01 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) May 22 18:06:01 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 18:06:01 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bb034cc00 x1400143561485277/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731661 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:06:01 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 22 18:06:13 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 18:06:13 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 18:06:13 oss2 kernel: LustreError: 13239:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810214c27400 x1400143561485287/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731673 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:06:13 oss2 kernel: LustreError: 13239:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 22 18:06:15 oss2 kernel: Lustre: 13284:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:06:15 oss2 kernel: Lustre: 13284:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message May 22 18:06:17 oss2 kernel: Lustre: 13284:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:06:23 oss2 kernel: Lustre: 13218:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:06:52 oss2 kernel: Lustre: 13171:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:09:12 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 18:09:12 oss2 kernel: LustreError: 13240:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105aebabc00 x1400143561485557/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731852 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:09:29 oss2 kernel: Lustre: 13272:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:09:29 oss2 kernel: Lustre: 13272:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages May 22 18:09:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 18:09:31 oss2 kernel: LustreError: 13180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102d6936400 x1400143561485906/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731871 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:09:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) May 22 18:09:37 oss2 kernel: LustreError: 13200:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102a0228800 x1400143561485911/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731877 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:09:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 18:09:50 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 18:09:50 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102e7eb9c00 x1400143561485923/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731890 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:09:50 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 22 18:09:56 oss2 kernel: Lustre: 13286:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:09:56 oss2 kernel: Lustre: 13286:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages May 22 18:10:11 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 18:10:11 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 18:10:11 oss2 kernel: LustreError: 13188:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b1fbba000 x1400143561485943/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337731911 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:10:11 oss2 kernel: LustreError: 13188:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 18:11:38 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 18:11:38 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages May 22 18:11:58 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 18:11:58 oss2 kernel: LustreError: 13195:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105e6a92c00 x1400143561486082/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337732018 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:27:43 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 18:27:43 oss2 kernel: Lustre: Skipped 1 previous similar message May 22 18:31:07 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 18:31:07 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 18:31:07 oss2 kernel: LustreError: 13175:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106e7b2f800 x1400143561487623/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337733167 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:31:07 oss2 kernel: LustreError: 13175:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 22 18:53:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 18:53:54 oss2 kernel: LustreError: Skipped 4 previous similar messages May 22 18:53:54 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061101a450 x1400143561489615/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337734534 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:53:54 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages May 22 18:54:33 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 18:54:33 oss2 kernel: LustreError: Skipped 6 previous similar messages May 22 18:54:33 oss2 kernel: LustreError: 13213:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105d6d2f400 x1400143561489670/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337734573 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:54:33 oss2 kernel: LustreError: 13213:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 22 18:54:34 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) May 22 18:54:34 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 18:54:34 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c9ec50 x1400143561489679/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337734574 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:54:34 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 22 18:55:12 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 18:55:12 oss2 kernel: LustreError: Skipped 3 previous similar messages May 22 18:55:12 oss2 kernel: LustreError: 13236:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a2db04800 x1400143561489725/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337734612 ref 1 fl Interpret:/0/0 rc -19/0 May 22 18:55:12 oss2 kernel: LustreError: 13236:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages May 22 19:04:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 19:04:03 oss2 kernel: LustreError: Skipped 6 previous similar messages May 22 19:04:03 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102f5c4ac00 x1400143561490599/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337735143 ref 1 fl Interpret:/0/0 rc -19/0 May 22 19:04:03 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 22 19:04:17 oss2 kernel: Lustre: 13267:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 5481adb7-ebd9-fe4d-2440-b1696f81d1eb reconnecting May 22 19:04:17 oss2 kernel: Lustre: 13267:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages May 22 19:12:43 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 19:12:43 oss2 kernel: Lustre: Skipped 5 previous similar messages May 22 19:12:54 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 19:12:54 oss2 kernel: Lustre: Skipped 2 previous similar messages May 22 19:13:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 19:13:08 oss2 kernel: LustreError: Skipped 6 previous similar messages May 22 19:13:08 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101a3292c00 x1400143561491270/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337735688 ref 1 fl Interpret:/0/0 rc -19/0 May 22 19:13:08 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 22 19:13:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 19:13:50 oss2 kernel: LustreError: Skipped 6 previous similar messages May 22 19:13:50 oss2 kernel: LustreError: 13289:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b8c63d400 x1400143561491322/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337735730 ref 1 fl Interpret:/0/0 rc -19/0 May 22 19:13:50 oss2 kernel: LustreError: 13289:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages May 22 21:26:28 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 21:26:28 oss2 kernel: Lustre: Skipped 2 previous similar messages May 22 21:28:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) May 22 21:28:20 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 21:28:20 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c87050 x1400143561501647/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337743800 ref 1 fl Interpret:/0/0 rc -19/0 May 22 21:28:20 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 21:28:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 21:28:38 oss2 kernel: LustreError: 13265:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061e80f000 x1400143561501657/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337743818 ref 1 fl Interpret:/0/0 rc -19/0 May 22 21:32:28 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 22 21:32:28 oss2 kernel: Lustre: Skipped 5 previous similar messages May 22 21:34:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) May 22 21:34:02 oss2 kernel: LustreError: Skipped 1 previous similar message May 22 21:34:02 oss2 kernel: LustreError: 13215:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b54985c00 x1400143561502001/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337744142 ref 1 fl Interpret:/0/0 rc -19/0 May 22 21:34:02 oss2 kernel: LustreError: 13215:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message May 22 21:40:15 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 22 21:40:15 oss2 kernel: LustreError: Skipped 2 previous similar messages May 22 21:40:15 oss2 kernel: LustreError: 13247:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101e185d800 x1400143561502285/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337744515 ref 1 fl Interpret:/0/0 rc -19/0 May 22 21:40:15 oss2 kernel: LustreError: 13247:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages May 23 00:40:04 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) May 23 00:40:04 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c10ef3c00 x1400143561510681/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1337755304 ref 1 fl Interpret:/0/0 rc -19/0 May 25 13:16:04 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5481adb7-ebd9-fe4d-2440-b1696f81d1eb (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 25 21:43:29 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 4265f9c3-59ef-bfaf-fdc0-bfdb81bf1f0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 25 21:43:29 oss2 kernel: Lustre: Skipped 5 previous similar messages May 25 21:52:06 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8103722cdc00 x1402967549833271/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1338004342 ref 1 fl Interpret:H/0/0 rc -107/0 May 25 21:52:06 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages May 26 02:33:07 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 4265f9c3-59ef-bfaf-fdc0-bfdb81bf1f0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 26 02:33:07 oss2 kernel: Lustre: Skipped 5 previous similar messages May 26 02:33:07 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 4265f9c3-59ef-bfaf-fdc0-bfdb81bf1f0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 26 02:33:07 oss2 kernel: Lustre: Skipped 1 previous similar message May 26 02:35:37 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810c372ad400 x1402967549852227/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1338021356 ref 1 fl Interpret:H/0/0 rc -107/0 May 26 02:35:37 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages May 28 20:51:25 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 4265f9c3-59ef-bfaf-fdc0-bfdb81bf1f0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 28 20:51:25 oss2 kernel: Lustre: Skipped 3 previous similar messages May 28 20:51:50 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 4265f9c3-59ef-bfaf-fdc0-bfdb81bf1f0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. May 28 20:51:50 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 8 14:17:14 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 238ed715-eec1-b584-20e4-3cc96695c373 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 8 14:17:14 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 8 16:18:18 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 204debc4-ee6e-e4fb-7610-2ac7a9b4adee (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 8 16:18:18 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 13 16:41:01 oss2 kernel: Lustre: 13252:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:41:01 oss2 kernel: Lustre: 13252:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 13 16:46:55 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810adc346780 Jun 13 16:46:55 oss2 kernel: LustreError: 12730:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810adc346780 Jun 13 16:46:55 oss2 kernel: LustreError: 4446:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(12682) req@ffff81057a831000 x1403332264915272/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339627626 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 16:46:55 oss2 kernel: LustreError: 4446:0:(ost_handler.c:1073:ost_brw_write()) Skipped 8 previous similar messages Jun 13 16:46:55 oss2 kernel: Lustre: 4446:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 16:46:55 oss2 kernel: Lustre: 4446:0:(ost_handler.c:1224:ost_brw_write()) Skipped 14 previous similar messages Jun 13 16:46:55 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:46:55 oss2 kernel: Lustre: 13228:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Jun 13 16:50:03 oss2 kernel: Lustre: 13271:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:50:07 oss2 kernel: Lustre: 13245:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:59:08 oss2 kernel: LustreError: 12728:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8104cd01cd80 Jun 13 16:59:08 oss2 kernel: LustreError: 12728:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff8104cd01cd80 Jun 13 16:59:08 oss2 kernel: LustreError: 13374:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(11417) req@ffff810b2f927400 x1403332264916221/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339628358 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 16:59:08 oss2 kernel: Lustre: 13374:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 16:59:09 oss2 kernel: Lustre: 13218:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:59:09 oss2 kernel: Lustre: 13218:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 13 16:59:51 oss2 kernel: Lustre: 13288:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 16:59:51 oss2 kernel: Lustre: 13288:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 13 17:10:07 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 17:10:07 oss2 kernel: LustreError: Skipped 3 previous similar messages Jun 13 17:10:07 oss2 kernel: LustreError: 13199:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c93450 x1403332264917278/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629107 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:10:07 oss2 kernel: LustreError: 13199:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 13 17:11:02 oss2 kernel: Lustre: 13280:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 17:11:02 oss2 kernel: Lustre: 13280:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff8101956fc800; still busy with 1 active RPCs Jun 13 17:11:02 oss2 kernel: Lustre: 13280:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message Jun 13 17:11:02 oss2 kernel: LustreError: 13280:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810120652800 x1403332264917360/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339629162 ref 1 fl Interpret:/0/0 rc -16/0 Jun 13 17:11:02 oss2 kernel: LustreError: 13280:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 13 17:11:22 oss2 kernel: Lustre: 13251:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 17:13:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 17:13:38 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 13 17:13:38 oss2 kernel: LustreError: 13232:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b25363400 x1403332264917581/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629318 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:13:51 oss2 kernel: Lustre: 13192:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 17:14:34 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Jun 13 17:14:34 oss2 kernel: LustreError: 13206:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81068603c400 x1403332264917653/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629374 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:14:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 17:14:43 oss2 kernel: LustreError: Skipped 1 previous similar message Jun 13 17:14:43 oss2 kernel: LustreError: 13252:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810aea08e400 x1403332264917658/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629383 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:14:43 oss2 kernel: LustreError: 13252:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 13 17:14:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 17:14:54 oss2 kernel: LustreError: 13222:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c373a3c00 x1403332264917676/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629394 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:23:50 oss2 kernel: Lustre: 13177:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 17:23:50 oss2 kernel: Lustre: 13177:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 13 17:24:04 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Jun 13 17:24:04 oss2 kernel: LustreError: Skipped 1 previous similar message Jun 13 17:24:04 oss2 kernel: LustreError: 13184:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061f39b400 x1403332264918611/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339629944 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:24:04 oss2 kernel: LustreError: 13184:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 13 17:24:57 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 17:24:57 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Jun 13 17:24:57 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff8101956fc800; still busy with 1 active RPCs Jun 13 17:24:57 oss2 kernel: LustreError: 13195:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810719477000 x1403332264918701/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339629997 ref 1 fl Interpret:/0/0 rc -16/0 Jun 13 17:24:57 oss2 kernel: LustreError: 13195:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 13 17:24:58 oss2 kernel: LustreError: 13352:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff81039203fc00 x1403332264918673/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339629913 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 17:24:58 oss2 kernel: LustreError: 13352:0:(ost_handler.c:1064:ost_brw_write()) Skipped 4 previous similar messages Jun 13 17:25:01 oss2 kernel: LustreError: 12734:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff81061474a4c0 Jun 13 17:25:01 oss2 kernel: LustreError: 12734:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff81061474a4c0 Jun 13 17:25:01 oss2 kernel: Lustre: 13352:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 17:29:21 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:29:21 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 13 17:29:21 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:29:21 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 17:39:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 17:39:50 oss2 kernel: LustreError: Skipped 1 previous similar message Jun 13 17:39:50 oss2 kernel: LustreError: 13291:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b8fbf8800 x1403332264919422/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339630890 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:39:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Jun 13 17:39:54 oss2 kernel: LustreError: Skipped 3 previous similar messages Jun 13 17:39:54 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b0e38d800 x1403332264919435/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339630894 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:39:54 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 13 17:43:21 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:43:21 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 13 17:43:37 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:43:41 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:43:41 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 17:43:43 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:43:43 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 17:44:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 17:44:37 oss2 kernel: LustreError: Skipped 1 previous similar message Jun 13 17:44:37 oss2 kernel: LustreError: 13178:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106143f3050 x1403332264919713/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339631177 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:44:37 oss2 kernel: LustreError: 13178:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Jun 13 17:44:57 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Jun 13 17:44:57 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a91aa6000 x1403332264919731/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339631197 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:48:44 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:48:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 17:48:50 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 13 17:48:50 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b8aff6800 x1403332264919982/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339631430 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 17:48:50 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 13 17:48:55 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:48:55 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 17:52:51 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 17:52:51 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 13 18:00:55 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 18:00:55 oss2 kernel: LustreError: Skipped 4 previous similar messages Jun 13 18:00:55 oss2 kernel: LustreError: 13241:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102b153b400 x1403332264920629/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339632155 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 18:00:55 oss2 kernel: LustreError: 13241:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 13 18:04:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Jun 13 18:04:08 oss2 kernel: LustreError: Skipped 3 previous similar messages Jun 13 18:04:08 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611756c50 x1403332264920798/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339632348 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 18:04:08 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 13 18:04:19 oss2 kernel: Lustre: 13245:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:04:19 oss2 kernel: Lustre: 13245:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Jun 13 18:05:53 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:05:53 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 18:08:06 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:27:58 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:27:58 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 13 18:28:06 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:28:06 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 13 18:28:21 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:29:55 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 18:29:55 oss2 kernel: LustreError: Skipped 4 previous similar messages Jun 13 18:29:55 oss2 kernel: LustreError: 13191:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81025eda3400 x1403332264922152/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339633895 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 18:29:55 oss2 kernel: LustreError: 13191:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 13 18:31:48 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:31:48 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Jun 13 18:31:49 oss2 kernel: Lustre: 13197:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:31:49 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff81071edd5800; still busy with 1 active RPCs Jun 13 18:31:49 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810611682850 x1403332264922296/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339634009 ref 1 fl Interpret:/0/0 rc -16/0 Jun 13 18:31:49 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 13 18:31:49 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff81071edd5800; still busy with 1 active RPCs Jun 13 18:31:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8105f91333c0 Jun 13 18:31:49 oss2 kernel: LustreError: 12746:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8105f91333c0 Jun 13 18:31:49 oss2 kernel: LustreError: 4514:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8106a3d10c00 x1403332264922292/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339633940 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 18:31:49 oss2 kernel: Lustre: 4514:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 18:31:51 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 18:31:51 oss2 kernel: LustreError: Skipped 4 previous similar messages Jun 13 18:32:04 oss2 kernel: Lustre: 13256:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:32:04 oss2 kernel: Lustre: 13256:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 7 previous similar messages Jun 13 18:32:36 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:32:36 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Jun 13 18:33:01 oss2 kernel: Lustre: 13209:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:33:01 oss2 kernel: Lustre: 13264:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:35:18 oss2 kernel: Lustre: 13232:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 18:35:18 oss2 kernel: Lustre: 13232:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Jun 13 18:39:49 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8106024716c0 Jun 13 18:39:49 oss2 kernel: LustreError: 12733:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff8106024716c0 Jun 13 18:39:49 oss2 kernel: LustreError: 13326:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(9348) req@ffff8101d0be1000 x1403332264923014/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1339634408 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 18:39:49 oss2 kernel: Lustre: 13326:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 18:42:40 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:42:40 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 13 18:43:12 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 18:43:12 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 13 18:51:49 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:06:35 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:44:23 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 19:44:23 oss2 kernel: LustreError: 13191:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81048cfb4c00 x1403332264926894/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339638363 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 19:44:23 oss2 kernel: LustreError: 13191:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Jun 13 19:48:30 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:48:30 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 13 19:48:50 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:48:51 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:49:01 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 19:58:39 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 19:58:39 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 13 19:58:39 oss2 kernel: LustreError: 13207:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106c0255c00 x1403332264928231/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639219 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 19:58:39 oss2 kernel: LustreError: 13207:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 13 19:59:04 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 19:59:04 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611fc9850 x1403332264928240/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639244 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 19:59:11 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Jun 13 19:59:11 oss2 kernel: LustreError: Skipped 4 previous similar messages Jun 13 19:59:11 oss2 kernel: LustreError: 13264:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810384dfac00 x1403332264928258/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639251 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 19:59:11 oss2 kernel: LustreError: 13264:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Jun 13 19:59:54 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 19:59:54 oss2 kernel: LustreError: 13287:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b32b39c00 x1403332264928290/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639294 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:00:17 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 20:00:17 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611971050 x1403332264928316/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639317 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:00:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 20:00:42 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 13 20:00:42 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810435ce6800 x1403332264928426/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639342 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:00:42 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 13 20:04:20 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 20:04:20 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 13 20:11:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 20:11:37 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810babf91c00 x1403332264928968/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339639997 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:33:55 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 20:33:55 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 13 20:52:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 20:52:38 oss2 kernel: LustreError: 13260:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c54050 x1403332264931746/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339642458 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:52:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 13 20:52:42 oss2 kernel: LustreError: 13281:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810301b9fc00 x1403332264931750/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339642462 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:53:22 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Jun 13 20:53:22 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 13 20:53:22 oss2 kernel: LustreError: 13283:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81041ab04800 x1403332264931823/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339642502 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 20:53:22 oss2 kernel: LustreError: 13283:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 13 20:53:31 oss2 kernel: Lustre: 13202:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 20:53:31 oss2 kernel: Lustre: 13202:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Jun 13 21:48:35 oss2 kernel: Lustre: 13261:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 21:48:35 oss2 kernel: Lustre: 13261:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Jun 13 22:08:20 oss2 kernel: Lustre: 13219:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 22:08:20 oss2 kernel: Lustre: 13219:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Jun 13 22:08:20 oss2 kernel: Lustre: 13219:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff8106029c2400; still busy with 1 active RPCs Jun 13 22:08:20 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81061a3f7c00 x1403332264938479/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339647000 ref 1 fl Interpret:/0/0 rc -16/0 Jun 13 22:08:20 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 13 22:08:20 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8104cd01c380 Jun 13 22:08:20 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8104cd01c380 Jun 13 22:08:20 oss2 kernel: LustreError: 7914:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff810c3ead6800 x1403332264938478/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339646922 ref 1 fl Interpret:/0/0 rc 0/0 Jun 13 22:08:20 oss2 kernel: Lustre: 7914:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 13 22:08:31 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 23:30:10 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 23:30:10 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 13 23:30:10 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 23:30:15 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 23:30:15 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 13 23:34:26 oss2 kernel: LustreError: 13170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810322c91c00 x1403332264954403/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339652091 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 13 23:34:26 oss2 kernel: LustreError: 13181:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81068c632c00 x1403332264954505/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339652091 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 13 23:34:26 oss2 kernel: LustreError: 13181:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23 previous similar messages Jun 13 23:34:26 oss2 kernel: LustreError: 13226:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b65eecc00 x1403332264954910/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339652091 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 13 23:34:26 oss2 kernel: LustreError: 13226:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 89 previous similar messages Jun 13 23:34:41 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 13 23:34:41 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 13 23:34:41 oss2 kernel: LustreError: 13255:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810658b63400 x1403332264954943/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339652181 ref 1 fl Interpret:/0/0 rc -19/0 Jun 13 23:34:49 oss2 kernel: Lustre: 13189:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 13 23:39:48 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 13 23:41:50 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8108702c1c00 x1403332264955109/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339652534 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 13 23:41:50 oss2 kernel: LustreError: 13235:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 14 00:22:21 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:22:21 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 00:22:40 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:22:40 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 00:26:45 oss2 kernel: LustreError: 13205:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8107182f6c00 x1403332264958475/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339655221 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 00:26:45 oss2 kernel: LustreError: 13205:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 78 previous similar messages Jun 14 00:32:15 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:32:33 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:32:33 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 00:32:50 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8107355bb400 x1403332264959138/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339655587 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 00:32:50 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 107 previous similar messages Jun 14 00:43:05 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 97fc847a-a3ef-2f2c-c884-6ec2d73ba7dc (at 192.168.2.1@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:43:05 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 97fc847a-a3ef-2f2c-c884-6ec2d73ba7dc (at 192.168.2.1@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:43:05 oss2 kernel: Lustre: Skipped 14 previous similar messages Jun 14 00:43:09 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 32e52cd8-2a94-6f4d-9dde-30dcd4f9c7df (at 192.168.2.2@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 00:43:09 oss2 kernel: Lustre: Skipped 9 previous similar messages Jun 14 01:15:18 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 01:15:18 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 14 01:15:18 oss2 kernel: LustreError: 13247:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810153096400 x1403332264962222/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339658218 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 01:15:18 oss2 kernel: LustreError: 13247:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 60 previous similar messages Jun 14 01:15:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Jun 14 01:15:28 oss2 kernel: LustreError: 13276:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810aea08e000 x1403332264962238/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339658228 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 01:19:35 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 01:19:35 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 01:19:36 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 01:19:36 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 01:43:15 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8105f125acc0 Jun 14 01:43:15 oss2 kernel: LustreError: 12740:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff8105f125acc0 Jun 14 01:43:15 oss2 kernel: LustreError: 13422:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(19765) req@ffff810151854c00 x1403332264963943/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339659804 ref 1 fl Interpret:/0/0 rc 0/0 Jun 14 01:43:15 oss2 kernel: Lustre: 13422:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 14 01:43:32 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 01:43:32 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Jun 14 01:44:14 oss2 kernel: Lustre: 13262:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 01:44:14 oss2 kernel: Lustre: 13262:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 14 01:44:37 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 01:44:37 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 01:44:37 oss2 kernel: LustreError: 13277:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106115b0050 x1403332264964093/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339659977 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 01:44:37 oss2 kernel: LustreError: 13277:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 01:48:05 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 01:48:05 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 01:48:05 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 14 01:48:14 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 01:48:14 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 14 01:56:26 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:02:24 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c45cc9e1-7356-5ab8-0b4c-ec3a10deecce (at 192.168.2.1@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:02:24 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 02:07:43 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:07:43 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 14 02:21:35 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 02:21:35 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 14 02:21:35 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810413877000 x1403332264965645/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339662195 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 02:21:35 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 14 02:21:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 02:21:38 oss2 kernel: LustreError: 13180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106c544a000 x1403332264965649/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339662198 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 02:25:25 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:25:25 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 02:28:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 02:28:03 oss2 kernel: LustreError: Skipped 3 previous similar messages Jun 14 02:28:03 oss2 kernel: LustreError: 13238:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810625013800 x1403332264966093/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339662583 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 02:28:03 oss2 kernel: LustreError: 13238:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 14 02:29:51 oss2 kernel: Lustre: 13199:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 02:30:27 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810259c467c0 Jun 14 02:30:27 oss2 kernel: LustreError: 12741:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff810259c467c0 Jun 14 02:30:27 oss2 kernel: LustreError: 7806:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(17247) req@ffff810821337400 x1403332264966331/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1339662647 ref 1 fl Interpret:/0/0 rc 0/0 Jun 14 02:30:27 oss2 kernel: Lustre: 7806:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 14 02:33:49 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:33:49 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 14 02:41:17 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 02:41:17 oss2 kernel: LustreError: 13275:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bb329bc00 x1403332264967120/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339663377 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 02:41:35 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Jun 14 02:41:35 oss2 kernel: LustreError: 13223:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611e17050 x1403332264967129/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339663395 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 02:47:48 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 27a3e0c8-86ce-9805-5207-e5e20fe649e5 (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 02:47:48 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 03:00:12 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:00:12 oss2 kernel: Lustre: Skipped 29 previous similar messages Jun 14 03:02:42 oss2 kernel: LustreError: 4510:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8107087bf400 x1403332264968765/t0 o4->@:0/0 lens 448/0 e 0 to 0 dl 1339664605 ref 1 fl Interpret:/0/0 rc -107/0 Jun 14 03:02:42 oss2 kernel: LustreError: 4510:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 03:14:46 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:14:46 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 03:21:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 03:21:21 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 03:21:21 oss2 kernel: LustreError: 13257:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611e12850 x1403332264971972/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339665781 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 03:21:21 oss2 kernel: LustreError: 13257:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 84 previous similar messages Jun 14 03:21:39 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 03:21:39 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 03:21:39 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bea07ec00 x1403332264971997/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339665799 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 03:21:39 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 03:40:37 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 161c181b-261b-78b7-69ad-fb8987dea933 (at 192.168.2.4@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:40:37 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 03:40:37 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 161c181b-261b-78b7-69ad-fb8987dea933 (at 192.168.2.4@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:40:37 oss2 kernel: Lustre: Skipped 14 previous similar messages Jun 14 03:46:30 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 8a9ec60e-ed4a-3bbd-a084-fe3cd9c0675f (at 192.168.2.1@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:46:30 oss2 kernel: Lustre: Skipped 14 previous similar messages Jun 14 03:51:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 03:51:03 oss2 kernel: LustreError: 13258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8107f40ba000 x1403332264974672/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339667563 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 03:56:15 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client d4d00597-1e26-0240-29ac-8e62fe45363c (at 192.168.2.4@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 03:56:15 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 04:01:03 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 04:01:03 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 14 04:07:39 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 04:07:39 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 04:07:39 oss2 kernel: LustreError: 13207:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810322c91800 x1403332264975755/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339668559 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:07:39 oss2 kernel: LustreError: 13207:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 04:07:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Jun 14 04:07:42 oss2 kernel: LustreError: 13208:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81032bebfc00 x1403332264975771/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339668562 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:10:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Jun 14 04:10:45 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 04:10:45 oss2 kernel: LustreError: 13286:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81091dc5c800 x1403332264976083/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339668745 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:10:45 oss2 kernel: LustreError: 13286:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 04:10:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Jun 14 04:10:48 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 04:10:48 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81057015f400 x1403332264976084/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339668748 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:10:48 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 04:16:50 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 04:16:50 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 04:33:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Jun 14 04:33:03 oss2 kernel: LustreError: 13293:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810219103800 x1403332264977651/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339670083 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:37:13 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 04:37:13 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 04:37:40 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 04:37:40 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 14 04:42:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 04:42:08 oss2 kernel: LustreError: 13285:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810504061000 x1403332264978112/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339670628 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:42:39 oss2 kernel: Lustre: 13229:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 04:42:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 04:42:49 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 14 04:42:49 oss2 kernel: LustreError: 13203:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810414666000 x1403332264978166/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339670669 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:42:49 oss2 kernel: LustreError: 13203:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 14 04:51:28 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 04:53:30 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 04:53:30 oss2 kernel: LustreError: Skipped 6 previous similar messages Jun 14 04:53:30 oss2 kernel: LustreError: 13288:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81024d39b400 x1403332264979227/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339671310 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:53:30 oss2 kernel: LustreError: 13288:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jun 14 04:53:41 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Jun 14 04:53:41 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 04:53:41 oss2 kernel: LustreError: 13249:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81035520d800 x1403332264979248/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339671321 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 04:53:41 oss2 kernel: LustreError: 13249:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 05:12:20 oss2 kernel: Lustre: 13213:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 05:36:50 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 05:36:50 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 05:36:59 oss2 kernel: LustreError: 13376:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810194beac00 x1403332264982841/t0 o4->@:0/0 lens 448/0 e 0 to 0 dl 1339673862 ref 1 fl Interpret:/0/0 rc -107/0 Jun 14 05:37:00 oss2 kernel: LustreError: 13214:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810882ff4800 x1403332264983042/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339673839 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 05:37:00 oss2 kernel: LustreError: 13214:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 44 previous similar messages Jun 14 05:56:33 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 05:56:33 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 05:56:41 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 05:56:41 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 05:56:58 oss2 kernel: LustreError: 13184:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8104a06cd400 x1403332264984444/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339675035 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 05:56:58 oss2 kernel: LustreError: 13184:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9 previous similar messages Jun 14 05:56:58 oss2 kernel: LustreError: 13244:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8105bbb86000 x1403332264984480/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339675034 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 05:56:58 oss2 kernel: LustreError: 13244:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Jun 14 05:56:59 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8106195a2800 x1403332264984496/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339675035 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 05:56:59 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages Jun 14 05:57:00 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b2288f000 x1403332264984669/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339675036 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 05:57:00 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar messages Jun 14 06:24:20 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 06:24:45 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 06:24:45 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 06:25:02 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810598877800 x1403332264987229/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339676718 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 06:25:02 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10 previous similar messages Jun 14 06:25:05 oss2 kernel: LustreError: 13253:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810555106800 x1403332264987264/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339676721 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 06:25:05 oss2 kernel: LustreError: 13253:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Jun 14 06:26:14 oss2 kernel: Lustre: 13206:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 06:29:33 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8102647d5c80 Jun 14 06:29:33 oss2 kernel: LustreError: 12744:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff8102647d5c80 Jun 14 06:29:33 oss2 kernel: LustreError: 4415:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(8859) req@ffff81060f8cc450 x1403332264987723/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1339676992 ref 1 fl Interpret:/0/0 rc 0/0 Jun 14 06:29:33 oss2 kernel: Lustre: 4415:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 14 06:31:32 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 06:31:44 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client bdc68653-85e3-848a-3776-6b2db6a95a3c (at 192.168.2.2@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 06:31:44 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 07:11:19 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 07:11:19 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810448dd8000 x1403332264990379/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339679579 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 07:11:19 oss2 kernel: LustreError: 13210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 44 previous similar messages Jun 14 07:19:09 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client e8d15e68-265e-59d1-46a9-da71216791c6 (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 07:19:09 oss2 kernel: Lustre: Skipped 24 previous similar messages Jun 14 07:36:29 oss2 kernel: Lustre: 13263:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 07:36:29 oss2 kernel: Lustre: 13263:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 14 07:36:29 oss2 kernel: Lustre: 13295:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 07:36:29 oss2 kernel: Lustre: 13295:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Jun 14 07:51:50 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 07:51:50 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 14 07:51:54 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 07:51:54 oss2 kernel: Lustre: Skipped 4 previous similar messages Jun 14 07:55:09 oss2 kernel: LustreError: 13175:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810ad89e0400 x1403332264993486/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339682136 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 07:55:09 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b86497400 x1403332264993580/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1339682136 ref 1 fl Interpret:H/0/0 rc -107/0 Jun 14 07:55:09 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 21 previous similar messages Jun 14 08:02:47 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 00ba70ef-8d6b-4e23-bbb6-45ab629273aa (at 192.168.2.3@o2ib) in 213 seconds. I think it's dead, and I am evicting it. Jun 14 08:05:20 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:05:20 oss2 kernel: Lustre: Skipped 26 previous similar messages Jun 14 08:08:46 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 3f6a01f6-dc98-6104-9590-de98f661ae5a (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:08:46 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 14 08:16:01 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:16:01 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 14 08:16:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Jun 14 08:16:02 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810322c91800 x1403332264995457/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339683462 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 08:16:02 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 68 previous similar messages Jun 14 08:17:51 oss2 kernel: Lustre: 13219:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 08:17:51 oss2 kernel: Lustre: 13177:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff8106e1b3b800; still busy with 1 active RPCs Jun 14 08:17:51 oss2 kernel: LustreError: 13177:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810192617c00 x1403332264995616/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339683571 ref 1 fl Interpret:/0/0 rc -16/0 Jun 14 08:17:51 oss2 kernel: LustreError: 4571:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8105be87bc00 x1403332264995612/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 0 to 0 dl 1339683478 ref 1 fl Interpret:/0/0 rc 0/0 Jun 14 08:17:51 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff810b40b9d680 Jun 14 08:17:51 oss2 kernel: LustreError: 12745:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff810b40b9d680 Jun 14 08:17:51 oss2 kernel: Lustre: 4571:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 14 08:18:05 oss2 kernel: Lustre: 13264:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 08:18:05 oss2 kernel: Lustre: 13264:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Jun 14 08:20:37 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 452997b4-6247-1961-8bab-4756da2df637 (at 192.168.2.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:20:37 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 08:21:54 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:21:54 oss2 kernel: Lustre: Skipped 26 previous similar messages Jun 14 08:22:10 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 08:22:10 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 14 08:58:41 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 08:58:41 oss2 kernel: LustreError: 13250:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102214c8c00 x1403332264997757/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339686021 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 08:59:13 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Jun 14 08:59:13 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 08:59:13 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a00950c00 x1403332264997780/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339686053 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 08:59:13 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 08:59:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 08:59:24 oss2 kernel: LustreError: 13279:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81051fea1800 x1403332264997804/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339686064 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 08:59:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Jun 14 08:59:42 oss2 kernel: LustreError: Skipped 5 previous similar messages Jun 14 08:59:42 oss2 kernel: LustreError: 13274:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3f401c00 x1403332264997821/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1339686082 ref 1 fl Interpret:/0/0 rc -19/0 Jun 14 08:59:42 oss2 kernel: LustreError: 13274:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 09:01:14 oss2 kernel: Lustre: 13214:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 09:01:14 oss2 kernel: Lustre: 13214:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 84df806e-51dd-3415-cf61-1b63d3cd042f@192.168.11.19@o2ib to 0xffff810c35b9ba00; still busy with 1 active RPCs Jun 14 09:01:14 oss2 kernel: LustreError: 13214:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810448dd8800 x1403332264997947/t0 o8->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 368/264 e 0 to 0 dl 1339686174 ref 1 fl Interpret:/0/0 rc -16/0 Jun 14 09:01:14 oss2 kernel: LustreError: 13214:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Jun 14 09:01:15 oss2 kernel: LustreError: 7806:0:(ost_handler.c:1064:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8107cca10c00 x1403332264997946/t0 o4->84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID:0/0 lens 448/416 e 1 to 0 dl 1339686099 ref 1 fl Interpret:/0/0 rc 0/0 Jun 14 09:01:15 oss2 kernel: LustreError: 12734:0:(events.c:381:server_bulk_callback()) event type 4, status -61, desc ffff8101d9b22e80 Jun 14 09:01:15 oss2 kernel: LustreError: 12734:0:(events.c:381:server_bulk_callback()) event type 2, status -61, desc ffff8101d9b22e80 Jun 14 09:01:15 oss2 kernel: Lustre: 7806:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST000d: ignoring bulk IO comm error with 84df806e-51dd-3415-cf61-1b63d3cd042f@NET_0x50000c0a80b13_UUID id 12345-192.168.11.19@o2ib - client will retry Jun 14 09:01:21 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 84df806e-51dd-3415-cf61-1b63d3cd042f reconnecting Jun 14 09:01:58 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 14a351a5-8d6f-ea10-716f-31f4cc7f6420 (at 192.168.2.2@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 09:05:11 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 84df806e-51dd-3415-cf61-1b63d3cd042f (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 09:05:11 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 14 09:28:48 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 6286a63e-49d9-d7d1-8330-586373013804 (at 192.168.2.1@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 09:28:48 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 14 10:15:06 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client ab4cd8f6-b26c-3b0d-0e3e-f25bd1f53bb3 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 14 10:15:06 oss2 kernel: Lustre: Skipped 23 previous similar messages Jun 15 13:17:22 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jun 15 13:17:25 oss2 kernel: ib_srp: ASYNC event= 17 on device= mlx4_0 Jun 15 13:21:04 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client e0d34ac4-09f0-4ef6-5016-56569f54049f (at 192.168.2.85@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 15 13:21:04 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 20 09:35:06 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 5088d236-60e1-e1b0-d72e-fecc841a526d (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 20 09:35:06 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 20 09:35:06 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 5088d236-60e1-e1b0-d72e-fecc841a526d (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 20 09:35:06 oss2 kernel: Lustre: Skipped 1 previous similar message Jun 20 14:32:22 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 3b9c1139-340c-b469-0515-ee5089112c40 (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 20 14:32:22 oss2 kernel: Lustre: Skipped 3 previous similar messages Jun 25 15:29:42 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 261cb450-41d8-dfc0-3620-6479cb5632e6 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 25 15:29:42 oss2 kernel: Lustre: Skipped 5 previous similar messages Jun 25 15:29:42 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 261cb450-41d8-dfc0-3620-6479cb5632e6 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 25 15:29:42 oss2 kernel: Lustre: Skipped 2 previous similar messages Jun 25 15:57:43 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client 205d0a7d-c1fa-66a2-1465-a0a095dccf0c (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jun 25 15:57:43 oss2 kernel: Lustre: Skipped 2 previous similar messages Jul 5 01:38:49 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 09:15:16 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 11:04:06 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 12:29:16 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 13:27:57 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 13:45:40 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 13:50:13 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 13:56:38 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:19:10 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:19:10 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 5 14:19:10 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:21:55 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:21:58 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:22:50 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:23:47 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 2 Jul 5 14:23:47 oss2 kernel: usb 1-5: failed to restore interface 0 altsetting 0 (error=-71) Jul 5 14:23:47 oss2 kernel: usb 1-5: USB disconnect, address 2 Jul 5 14:23:47 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 3 Jul 5 14:23:47 oss2 kernel: usb 1-5: configuration #1 chosen from 1 choice Jul 5 14:23:47 oss2 kernel: scsi11 : SCSI emulation for USB Mass Storage devices Jul 5 14:23:47 oss2 kernel: usb-storage: device found at 3 Jul 5 14:23:47 oss2 kernel: usb-storage: waiting for device to settle before scanning Jul 5 14:23:47 oss2 kernel: scsi12 : SCSI emulation for USB Mass Storage devices Jul 5 14:23:47 oss2 kernel: usb-storage: device found at 3 Jul 5 14:23:47 oss2 kernel: usb-storage: waiting for device to settle before scanning Jul 5 14:23:47 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input3 Jul 5 14:23:47 oss2 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Jul 5 14:23:47 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input4 Jul 5 14:23:47 oss2 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Jul 5 14:23:52 oss2 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Jul 5 14:23:52 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Jul 5 14:23:52 oss2 kernel: Vendor: PepperC Model: Virtual Disc 2 Rev: 0.01 Jul 5 14:23:52 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Jul 5 14:23:52 oss2 kernel: sr1: scsi-1 drive Jul 5 14:23:52 oss2 kernel: sr 11:0:0:0: Attached scsi CD-ROM sr1 Jul 5 14:23:52 oss2 kernel: sr 11:0:0:0: Attached scsi generic sg2 type 5 Jul 5 14:23:52 oss2 kernel: sr2: scsi-1 drive Jul 5 14:23:52 oss2 kernel: sr 12:0:0:0: Attached scsi CD-ROM sr2 Jul 5 14:23:52 oss2 kernel: sr 12:0:0:0: Attached scsi generic sg3 type 5 Jul 5 14:23:52 oss2 kernel: usb-storage: device scan complete Jul 5 14:23:52 oss2 kernel: usb-storage: device scan complete Jul 6 22:33:17 oss2 kernel: Lustre: 13279:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1acfc793-ec3c-f95a-e723-e6e249937c9b reconnecting Jul 11 17:12:03 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client a738d26f-8444-8bb5-9a1a-ad3366b74420 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 11 17:12:03 oss2 kernel: Lustre: Skipped 5 previous similar messages Jul 11 17:12:03 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client a738d26f-8444-8bb5-9a1a-ad3366b74420 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 14 07:24:14 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 3 Jul 14 07:24:14 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 14 07:24:14 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 3 Jul 14 07:24:14 oss2 kernel: usb 1-5: device not accepting address 3, error -71 Jul 14 07:24:15 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 3 Jul 14 07:24:15 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 14 07:24:15 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 3 Jul 14 07:24:15 oss2 kernel: usb 1-5: device descriptor read/8, error -71 Jul 14 07:24:15 oss2 kernel: usb 1-5: device descriptor read/8, error -71 Jul 14 07:24:15 oss2 kernel: usb 1-5: USB disconnect, address 3 Jul 14 07:24:15 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 4 Jul 14 07:24:15 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 14 07:24:15 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 5 Jul 14 07:24:16 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 14 07:24:16 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 6 Jul 14 07:24:16 oss2 kernel: usb 1-5: device descriptor read/8, error -71 Jul 14 07:24:16 oss2 kernel: usb 1-5: device descriptor read/8, error -71 Jul 14 07:24:16 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 7 Jul 14 07:24:16 oss2 kernel: usb 1-5: device descriptor read/8, error -71 Jul 14 07:24:16 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Jul 15 12:16:18 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client cf7adcd0-011d-f02f-02de-1f43452e9758 (at 192.168.2.224@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 15 12:16:18 oss2 kernel: Lustre: Skipped 4 previous similar messages Jul 15 12:16:43 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client cf7adcd0-011d-f02f-02de-1f43452e9758 (at 192.168.2.224@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 15 12:16:49 oss2 kernel: LustreError: 15511:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b40876c00 x1403253384171697/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1342376225 ref 1 fl Interpret:H/0/0 rc -107/0 Jul 15 12:20:36 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client cf7adcd0-011d-f02f-02de-1f43452e9758 (at 192.168.2.224@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 15 12:20:36 oss2 kernel: Lustre: Skipped 3 previous similar messages Jul 15 12:20:48 oss2 kernel: LustreError: 18435:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810be03a0800 x1403253384189388/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1342376696 ref 1 fl Interpret:H/0/0 rc -107/0 Jul 15 12:20:48 oss2 kernel: LustreError: 18435:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages Jul 15 12:24:35 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client cf7adcd0-011d-f02f-02de-1f43452e9758 (at 192.168.2.224@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 15 12:24:47 oss2 kernel: LustreError: 13189:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8108d4867000 x1403253384209799/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1342376935 ref 1 fl Interpret:H/0/0 rc -107/0 Jul 15 12:28:34 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client cf7adcd0-011d-f02f-02de-1f43452e9758 (at 192.168.2.224@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 15 12:28:47 oss2 kernel: LustreError: 24890:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81030ff3a000 x1403253384210036/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1342377175 ref 1 fl Interpret:H/0/0 rc -107/0 Jul 18 13:38:45 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:38:48 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 18 13:39:05 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:39:08 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 18 13:39:15 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:39:28 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 18 13:39:47 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:39:49 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 18 13:43:09 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:43:36 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 18 13:44:01 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:44:03 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 18 13:44:20 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:44:23 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 18 13:44:30 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:44:50 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 18 13:45:10 oss2 kernel: igb: eth0 NIC Link is Down Jul 18 13:45:12 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 21 16:31:36 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 78207ac0-d109-186c-2139-9cd27020f662 (at 192.168.2.81@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 21 16:32:01 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 78207ac0-d109-186c-2139-9cd27020f662 (at 192.168.2.81@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 21 16:32:01 oss2 kernel: Lustre: Skipped 2 previous similar messages Jul 21 16:32:32 oss2 kernel: LustreError: 13426:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8108fa708400 x1401530660075837/t0 o3->@:0/0 lens 448/0 e 0 to 0 dl 1342909958 ref 1 fl Interpret:/0/0 rc -107/0 Jul 22 18:23:15 oss2 kernel: Lustre: lustre-OST000d: slow setattr 38s due to heavy IO load Jul 22 18:23:26 oss2 kernel: Lustre: lustre-OST0009: slow setattr 39s due to heavy IO load Jul 22 18:23:26 oss2 kernel: Lustre: lustre-OST0009: slow setattr 52s due to heavy IO load Jul 23 13:56:57 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 91b27551-aa6b-ee73-a82a-91975f4db639 (at 192.168.2.90@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 23 13:56:57 oss2 kernel: Lustre: Skipped 1 previous similar message Jul 23 13:57:02 oss2 kernel: LustreError: 18430:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff8107034b0c00 x1401530810111974/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1343073438 ref 1 fl Interpret:H/0/0 rc -107/0 Jul 23 13:57:02 oss2 kernel: LustreError: 18430:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Jul 27 15:46:45 oss2 kernel: Lustre: 18454:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 6cf0bb0c-d97e-c225-8593-ba90311f1627 reconnecting Jul 27 15:46:45 oss2 kernel: Lustre: 18454:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 7 previous similar messages Jul 27 16:03:07 oss2 kernel: igb: eth0 NIC Link is Down Jul 27 16:03:23 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 27 16:03:38 oss2 kernel: igb: eth0 NIC Link is Down Jul 27 16:03:41 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 27 23:03:40 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 1acfc793-ec3c-f95a-e723-e6e249937c9b (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 27 23:03:40 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 1acfc793-ec3c-f95a-e723-e6e249937c9b (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 27 23:03:40 oss2 kernel: Lustre: Skipped 2 previous similar messages Jul 27 23:04:01 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 1acfc793-ec3c-f95a-e723-e6e249937c9b (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 27 23:04:01 oss2 kernel: Lustre: Skipped 1 previous similar message Jul 30 12:32:52 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 12:39:59 oss2 kernel: Lustre: 18488:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: af1d878b-8a85-4b0d-c694-fc17dba8fd97 reconnecting Jul 30 12:39:59 oss2 kernel: Lustre: 18488:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Jul 30 13:31:43 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 30 13:32:07 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:32:10 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 13:32:27 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:32:30 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 13:32:37 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:32:48 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 30 13:33:07 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:33:10 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 13:52:06 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:54:53 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 30 13:55:18 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:55:20 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 13:55:37 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:55:40 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 13:55:48 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:56:02 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 30 13:56:21 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 13:56:24 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 15:50:35 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 15:50:37 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 15:50:55 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 15:50:57 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 30 15:51:06 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 15:51:20 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jul 30 15:51:39 oss2 kernel: igb: eth0 NIC Link is Down Jul 30 15:51:42 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jul 31 12:46:48 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client b2dcae38-f5b2-4aab-d811-162eea634b50 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 31 13:06:22 oss2 kernel: Lustre: lustre-OST0015: haven't heard from client bf6dafcb-87fb-0f6b-89b7-168fc9f91648 (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 31 13:06:22 oss2 kernel: Lustre: Skipped 5 previous similar messages Jul 31 13:48:59 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 49ac1c5c-16a3-47aa-8693-f20a679a3f7a (at 192.168.11.18@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jul 31 13:48:59 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 4 17:29:29 oss2 kernel: Lustre: 18416:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 9cd2d76d-dfad-0d95-0874-1bbbceaaf8b3 reconnecting Aug 4 17:29:29 oss2 kernel: Lustre: 18416:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 00:01:22 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:01:22 oss2 kernel: LustreError: Skipped 5 previous similar messages Aug 8 00:01:22 oss2 kernel: LustreError: 13295:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810691659800 x1398083060540885/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405782 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:22 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 00:01:22 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 00:01:22 oss2 kernel: LustreError: 10821:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81039e8c3c00 x1409233690709442/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405782 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:22 oss2 kernel: LustreError: 10821:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 00:01:28 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:01:28 oss2 kernel: LustreError: 10819:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061167f850 x1409048085718193/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405788 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:29 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:01:29 oss2 kernel: LustreError: Skipped 20 previous similar messages Aug 8 00:01:29 oss2 kernel: LustreError: 25858:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810aea08ec00 x1409229534569727/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405789 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:29 oss2 kernel: LustreError: 25858:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20 previous similar messages Aug 8 00:01:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:01:31 oss2 kernel: LustreError: Skipped 127 previous similar messages Aug 8 00:01:31 oss2 kernel: LustreError: 10835:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101ae66b800 x1409231356028863/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405791 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:31 oss2 kernel: LustreError: 10835:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 127 previous similar messages Aug 8 00:01:35 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 00:01:35 oss2 kernel: LustreError: Skipped 409 previous similar messages Aug 8 00:01:35 oss2 kernel: LustreError: 15560:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810998f2ac00 x1409233632744940/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405795 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:35 oss2 kernel: LustreError: 15560:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 409 previous similar messages Aug 8 00:01:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 00:01:43 oss2 kernel: LustreError: Skipped 597 previous similar messages Aug 8 00:01:43 oss2 kernel: LustreError: 13231:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810866561000 x1409231369729176/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405803 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:43 oss2 kernel: LustreError: 13231:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 597 previous similar messages Aug 8 00:01:59 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 00:01:59 oss2 kernel: LustreError: Skipped 1875 previous similar messages Aug 8 00:01:59 oss2 kernel: LustreError: 24898:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81055bd79c00 x1406603304557438/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405819 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:01:59 oss2 kernel: LustreError: 24898:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1875 previous similar messages Aug 8 00:02:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 00:02:31 oss2 kernel: LustreError: Skipped 3283 previous similar messages Aug 8 00:02:31 oss2 kernel: LustreError: 18437:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bbea02400 x1409234380258331/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405851 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:02:31 oss2 kernel: LustreError: 18437:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3283 previous similar messages Aug 8 00:03:35 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:03:35 oss2 kernel: LustreError: Skipped 5089 previous similar messages Aug 8 00:03:35 oss2 kernel: LustreError: 18425:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810519c71800 x1401526053097952/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344405915 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:03:35 oss2 kernel: LustreError: 18425:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5089 previous similar messages Aug 8 00:05:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 8 00:05:43 oss2 kernel: LustreError: Skipped 7801 previous similar messages Aug 8 00:05:43 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810685a6d400 x1409231222858731/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344406043 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:05:43 oss2 kernel: LustreError: 13174:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7801 previous similar messages Aug 8 00:10:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:10:09 oss2 kernel: LustreError: Skipped 11484 previous similar messages Aug 8 00:10:09 oss2 kernel: LustreError: 13201:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b2dd33800 x1398083060541762/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344406309 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:10:09 oss2 kernel: LustreError: 13201:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 11484 previous similar messages Aug 8 00:18:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 00:18:45 oss2 kernel: LustreError: Skipped 16261 previous similar messages Aug 8 00:18:45 oss2 kernel: LustreError: 24882:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81076e565000 x1409048085719401/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344406825 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:18:45 oss2 kernel: LustreError: 24882:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 16261 previous similar messages Aug 8 00:28:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 8 00:28:45 oss2 kernel: LustreError: Skipped 26038 previous similar messages Aug 8 00:28:45 oss2 kernel: LustreError: 10864:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101c227e800 x1409229533524301/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344407425 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:28:45 oss2 kernel: LustreError: 10864:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26038 previous similar messages Aug 8 00:38:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 00:38:45 oss2 kernel: LustreError: Skipped 25238 previous similar messages Aug 8 00:38:45 oss2 kernel: LustreError: 10840:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063db97400 x1409233753366960/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344408025 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:38:45 oss2 kernel: LustreError: 10840:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25238 previous similar messages Aug 8 00:48:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 00:48:49 oss2 kernel: LustreError: Skipped 17349 previous similar messages Aug 8 00:48:49 oss2 kernel: LustreError: 25841:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611e0fc50 x1409229530379770/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344408629 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:48:49 oss2 kernel: LustreError: 25841:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17349 previous similar messages Aug 8 00:58:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 00:58:49 oss2 kernel: LustreError: Skipped 32233 previous similar messages Aug 8 00:58:49 oss2 kernel: LustreError: 18437:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810619237c50 x1409231351832996/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344409229 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 00:58:49 oss2 kernel: LustreError: 18437:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32233 previous similar messages Aug 8 01:08:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 01:08:49 oss2 kernel: LustreError: Skipped 19579 previous similar messages Aug 8 01:08:49 oss2 kernel: LustreError: 10811:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c89c50 x1409231231263403/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344409829 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:08:49 oss2 kernel: LustreError: 10811:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19579 previous similar messages Aug 8 01:14:28 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 2eb6d39a-36bc-798c-a122-2f59ee1d8b3d (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 01:14:28 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 8 01:14:28 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 2eb6d39a-36bc-798c-a122-2f59ee1d8b3d (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 01:14:28 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 01:18:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 01:18:49 oss2 kernel: LustreError: Skipped 25348 previous similar messages Aug 8 01:18:49 oss2 kernel: LustreError: 16666:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810263352400 x1409233699816779/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344410429 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:18:49 oss2 kernel: LustreError: 16666:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25348 previous similar messages Aug 8 01:26:38 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client 2eb6d39a-36bc-798c-a122-2f59ee1d8b3d (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 01:26:38 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 01:28:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 01:28:50 oss2 kernel: LustreError: Skipped 25511 previous similar messages Aug 8 01:28:50 oss2 kernel: LustreError: 18483:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a29e3c400 x1409232008173507/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344411030 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:28:50 oss2 kernel: LustreError: 18483:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25511 previous similar messages Aug 8 01:38:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 01:38:50 oss2 kernel: LustreError: Skipped 17363 previous similar messages Aug 8 01:38:50 oss2 kernel: LustreError: 18480:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b57dafc00 x1409229645723444/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344411630 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:38:50 oss2 kernel: LustreError: 18480:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17363 previous similar messages Aug 8 01:48:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 01:48:50 oss2 kernel: LustreError: Skipped 31320 previous similar messages Aug 8 01:48:50 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81097101cc00 x1409229520951281/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344412230 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:48:50 oss2 kernel: LustreError: 13187:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31320 previous similar messages Aug 8 01:59:16 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 01:59:16 oss2 kernel: LustreError: Skipped 20068 previous similar messages Aug 8 01:59:16 oss2 kernel: LustreError: 10870:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101d26b9800 x1398083060549927/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344412856 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 01:59:16 oss2 kernel: LustreError: 10870:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 20068 previous similar messages Aug 8 02:09:16 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 02:09:16 oss2 kernel: LustreError: Skipped 26921 previous similar messages Aug 8 02:09:16 oss2 kernel: LustreError: 25834:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81056dbbf400 x1401526078423124/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344413456 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:09:16 oss2 kernel: LustreError: 25834:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26921 previous similar messages Aug 8 02:19:16 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 02:19:16 oss2 kernel: LustreError: Skipped 24056 previous similar messages Aug 8 02:19:16 oss2 kernel: LustreError: 13172:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063e16d400 x1409231980928696/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344414056 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:19:16 oss2 kernel: LustreError: 13172:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 24056 previous similar messages Aug 8 02:29:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 02:29:20 oss2 kernel: LustreError: Skipped 17648 previous similar messages Aug 8 02:29:20 oss2 kernel: LustreError: 25852:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810277af8400 x1398083060552003/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344414660 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:29:20 oss2 kernel: LustreError: 25852:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17648 previous similar messages Aug 8 02:39:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 8 02:39:20 oss2 kernel: LustreError: Skipped 32220 previous similar messages Aug 8 02:39:20 oss2 kernel: LustreError: 15527:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611fbc050 x1398083060552739/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344415260 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:39:20 oss2 kernel: LustreError: 15527:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32220 previous similar messages Aug 8 02:49:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 02:49:20 oss2 kernel: LustreError: Skipped 19058 previous similar messages Aug 8 02:49:20 oss2 kernel: LustreError: 13241:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c88450 x1409233667419808/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344415860 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:49:20 oss2 kernel: LustreError: 13241:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19058 previous similar messages Aug 8 02:59:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 02:59:20 oss2 kernel: LustreError: Skipped 26617 previous similar messages Aug 8 02:59:20 oss2 kernel: LustreError: 13283:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106f06f5000 x1409233721773223/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344416460 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 02:59:20 oss2 kernel: LustreError: 13283:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26617 previous similar messages Aug 8 03:09:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 03:09:21 oss2 kernel: LustreError: Skipped 24819 previous similar messages Aug 8 03:09:21 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106cdb6b800 x1409230263321735/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344417061 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:09:21 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 24819 previous similar messages Aug 8 03:19:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 03:19:21 oss2 kernel: LustreError: Skipped 16901 previous similar messages Aug 8 03:19:21 oss2 kernel: LustreError: 13211:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611b12050 x1409234382365380/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344417661 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:19:21 oss2 kernel: LustreError: 13211:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 16901 previous similar messages Aug 8 03:29:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 03:29:21 oss2 kernel: LustreError: Skipped 32035 previous similar messages Aug 8 03:29:21 oss2 kernel: LustreError: 24894:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810855deb400 x1409301083455864/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344418261 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:29:21 oss2 kernel: LustreError: 24894:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32035 previous similar messages Aug 8 03:39:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 03:39:21 oss2 kernel: LustreError: Skipped 19703 previous similar messages Aug 8 03:39:21 oss2 kernel: LustreError: 18474:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b6c9ab800 x1409229526213709/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344418861 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:39:21 oss2 kernel: LustreError: 18474:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19703 previous similar messages Aug 8 03:49:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 03:49:21 oss2 kernel: LustreError: Skipped 26235 previous similar messages Aug 8 03:49:21 oss2 kernel: LustreError: 10856:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c03795800 x1409231127437934/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344419461 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:49:21 oss2 kernel: LustreError: 10856:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26235 previous similar messages Aug 8 03:59:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 03:59:24 oss2 kernel: LustreError: Skipped 24527 previous similar messages Aug 8 03:59:24 oss2 kernel: LustreError: 10835:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81060b2b8000 x1409243432516166/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344420064 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 03:59:24 oss2 kernel: LustreError: 10835:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 24527 previous similar messages Aug 8 04:09:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 04:09:24 oss2 kernel: LustreError: Skipped 17842 previous similar messages Aug 8 04:09:24 oss2 kernel: LustreError: 15527:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810230af8000 x1409231309023095/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344420664 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:09:24 oss2 kernel: LustreError: 15527:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17842 previous similar messages Aug 8 04:19:24 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 04:19:24 oss2 kernel: LustreError: Skipped 31215 previous similar messages Aug 8 04:19:24 oss2 kernel: LustreError: 13289:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8107b717d400 x1409229569349591/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344421264 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:19:24 oss2 kernel: LustreError: 13289:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31215 previous similar messages Aug 8 04:29:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 04:29:42 oss2 kernel: LustreError: Skipped 19582 previous similar messages Aug 8 04:29:42 oss2 kernel: LustreError: 18487:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061189e850 x1398083060560384/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344421882 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:29:42 oss2 kernel: LustreError: 18487:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19582 previous similar messages Aug 8 04:39:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 04:39:42 oss2 kernel: LustreError: Skipped 27648 previous similar messages Aug 8 04:39:42 oss2 kernel: LustreError: 13224:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061990dc00 x1409230263481895/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344422482 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:39:42 oss2 kernel: LustreError: 13224:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27648 previous similar messages Aug 8 04:49:42 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 04:49:42 oss2 kernel: LustreError: Skipped 23411 previous similar messages Aug 8 04:49:42 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c3f95f800 x1409234382525652/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344423082 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:49:42 oss2 kernel: LustreError: 13185:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23411 previous similar messages Aug 8 04:59:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 8 04:59:43 oss2 kernel: LustreError: Skipped 17601 previous similar messages Aug 8 04:59:43 oss2 kernel: LustreError: 13287:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106115a7450 x1409231276435217/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344423683 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 04:59:43 oss2 kernel: LustreError: 13287:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17601 previous similar messages Aug 8 05:09:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 8 05:09:43 oss2 kernel: LustreError: Skipped 32353 previous similar messages Aug 8 05:09:43 oss2 kernel: LustreError: 10875:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81048c9d5800 x1403253371500428/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344424283 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:09:43 oss2 kernel: LustreError: 10875:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32353 previous similar messages Aug 8 05:19:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 05:19:43 oss2 kernel: LustreError: Skipped 18673 previous similar messages Aug 8 05:19:43 oss2 kernel: LustreError: 13170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81025fef5400 x1401514819902485/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344424883 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:19:43 oss2 kernel: LustreError: 13170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18673 previous similar messages Aug 8 05:29:43 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 05:29:43 oss2 kernel: LustreError: Skipped 27329 previous similar messages Aug 8 05:29:43 oss2 kernel: LustreError: 13259:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810123441000 x1409243328779513/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344425483 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:29:43 oss2 kernel: LustreError: 13259:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27329 previous similar messages Aug 8 05:39:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 05:39:44 oss2 kernel: LustreError: Skipped 24205 previous similar messages Aug 8 05:39:44 oss2 kernel: LustreError: 13284:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81031d479c00 x1409229520063758/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344426084 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:39:44 oss2 kernel: LustreError: 13284:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 24205 previous similar messages Aug 8 05:49:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 05:49:44 oss2 kernel: LustreError: Skipped 16760 previous similar messages Aug 8 05:49:44 oss2 kernel: LustreError: 13182:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810a52943800 x1409232007309113/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344426684 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:49:44 oss2 kernel: LustreError: 13182:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 16760 previous similar messages Aug 8 05:59:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 05:59:44 oss2 kernel: LustreError: Skipped 32324 previous similar messages Aug 8 05:59:44 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c10275000 x1409229591371811/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344427284 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 05:59:44 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32324 previous similar messages Aug 8 06:09:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 06:09:44 oss2 kernel: LustreError: Skipped 19541 previous similar messages Aug 8 06:09:44 oss2 kernel: LustreError: 15524:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810babf91400 x1409231371930055/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344427884 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 06:09:44 oss2 kernel: LustreError: 15524:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19541 previous similar messages Aug 8 06:19:44 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 06:19:44 oss2 kernel: LustreError: Skipped 26966 previous similar messages Aug 8 06:19:44 oss2 kernel: LustreError: 15518:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8104552c3400 x1409229520073925/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344428484 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 06:19:44 oss2 kernel: LustreError: 15518:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26966 previous similar messages Aug 8 06:29:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 06:29:50 oss2 kernel: LustreError: Skipped 23929 previous similar messages Aug 8 06:29:50 oss2 kernel: LustreError: 13189:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810136723000 x1409243432680408/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344429090 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 06:29:50 oss2 kernel: LustreError: 13189:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23929 previous similar messages Aug 8 06:39:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 06:39:50 oss2 kernel: LustreError: Skipped 17822 previous similar messages Aug 8 06:39:50 oss2 kernel: LustreError: 10868:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810b5994e000 x1409231334194292/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344429690 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 06:39:50 oss2 kernel: LustreError: 10868:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17822 previous similar messages Aug 8 06:49:50 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 06:49:50 oss2 kernel: LustreError: Skipped 31739 previous similar messages Aug 8 06:49:50 oss2 kernel: LustreError: 13293:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81035626b400 x1409231311131121/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344430290 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 06:49:50 oss2 kernel: LustreError: 13293:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31739 previous similar messages Aug 8 07:00:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 07:00:08 oss2 kernel: LustreError: Skipped 19078 previous similar messages Aug 8 07:00:08 oss2 kernel: LustreError: 13294:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103cdeea400 x1398083060571029/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344430908 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:00:08 oss2 kernel: LustreError: 13294:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19078 previous similar messages Aug 8 07:10:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 07:10:08 oss2 kernel: LustreError: Skipped 28266 previous similar messages Aug 8 07:10:08 oss2 kernel: LustreError: 24876:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105d7396800 x1409229526381994/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344431508 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:10:08 oss2 kernel: LustreError: 24876:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28266 previous similar messages Aug 8 07:20:08 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 07:20:08 oss2 kernel: LustreError: Skipped 22898 previous similar messages Aug 8 07:20:08 oss2 kernel: LustreError: 18474:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810873229800 x1409243327736495/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344432108 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:20:08 oss2 kernel: LustreError: 18474:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 22898 previous similar messages Aug 8 07:30:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 07:30:09 oss2 kernel: LustreError: Skipped 17545 previous similar messages Aug 8 07:30:09 oss2 kernel: LustreError: 24899:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109c681f400 x1409233659118269/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344432709 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:30:09 oss2 kernel: LustreError: 24899:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17545 previous similar messages Aug 8 07:40:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 07:40:09 oss2 kernel: LustreError: Skipped 32402 previous similar messages Aug 8 07:40:09 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611755850 x1409229550481113/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344433309 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:40:09 oss2 kernel: LustreError: 13228:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32402 previous similar messages Aug 8 07:50:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 07:50:09 oss2 kernel: LustreError: Skipped 18573 previous similar messages Aug 8 07:50:09 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063e803000 x1406603304745382/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344433909 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 07:50:09 oss2 kernel: LustreError: 13282:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18573 previous similar messages Aug 8 08:00:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 08:00:09 oss2 kernel: LustreError: Skipped 27932 previous similar messages Aug 8 08:00:09 oss2 kernel: LustreError: 13240:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810ba3a1b400 x1409233740807249/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344434509 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:00:09 oss2 kernel: LustreError: 13240:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27932 previous similar messages Aug 8 08:10:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 08:10:09 oss2 kernel: LustreError: Skipped 23695 previous similar messages Aug 8 08:10:09 oss2 kernel: LustreError: 18455:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c37c79000 x1409229529523853/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344435109 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:10:09 oss2 kernel: LustreError: 18455:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23695 previous similar messages Aug 8 08:20:09 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 08:20:09 oss2 kernel: LustreError: Skipped 16848 previous similar messages Aug 8 08:20:09 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810565863c00 x1409231356217125/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344435709 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:20:09 oss2 kernel: LustreError: 13204:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 16848 previous similar messages Aug 8 08:30:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 08:30:10 oss2 kernel: LustreError: Skipped 32242 previous similar messages Aug 8 08:30:10 oss2 kernel: LustreError: 25836:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81056dbbf800 x1409233866518191/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344436310 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:30:10 oss2 kernel: LustreError: 25836:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32242 previous similar messages Aug 8 08:40:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 08:40:10 oss2 kernel: LustreError: Skipped 19410 previous similar messages Aug 8 08:40:10 oss2 kernel: LustreError: 15517:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81021dbac800 x1409229577733649/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344436910 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:40:10 oss2 kernel: LustreError: 15517:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 19410 previous similar messages Aug 8 08:50:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 08:50:10 oss2 kernel: LustreError: Skipped 27560 previous similar messages Aug 8 08:50:10 oss2 kernel: LustreError: 15503:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bb3055000 x1409229519017240/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344437510 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 08:50:10 oss2 kernel: LustreError: 15503:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27560 previous similar messages Aug 8 09:00:20 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 09:00:20 oss2 kernel: LustreError: Skipped 23390 previous similar messages Aug 8 09:00:20 oss2 kernel: LustreError: 18414:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81056dbbf800 x1406058914807867/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344438120 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 09:00:20 oss2 kernel: LustreError: 18414:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23390 previous similar messages Aug 8 09:10:07 oss2 kernel: LDISKFS-fs warning (device dm-25): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:07 oss2 kernel: Aug 8 09:10:07 oss2 kernel: LDISKFS-fs warning (device dm-29): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:07 oss2 kernel: Aug 8 09:10:07 oss2 kernel: LDISKFS-fs warning (device dm-5): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:07 oss2 kernel: Aug 8 09:10:07 oss2 kernel: LDISKFS-fs warning (device dm-9): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:07 oss2 kernel: Aug 8 09:10:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 09:10:21 oss2 kernel: LustreError: Skipped 18171 previous similar messages Aug 8 09:10:21 oss2 kernel: LustreError: 18466:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81063f2e0400 x1409231276452057/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344438721 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 09:10:21 oss2 kernel: LustreError: 18466:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18171 previous similar messages Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-29): recovery complete Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Aug 8 09:10:50 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-29): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode Aug 8 09:10:50 oss2 kernel: Lustre: 4988:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0018, 258 recoverable clients, 0 delayed clients, last_rcvd 47256260581 Aug 8 09:10:50 oss2 kernel: JBD: barrier-based sync failed on dm-29-8 - disabling barriers Aug 8 09:10:50 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:50 oss2 kernel: Lustre: lustre-OST0018: Now serving lustre-OST0018 on /dev/mapper/ost_lustre_24 with recovery enabled Aug 8 09:10:50 oss2 kernel: Lustre: lustre-OST0018: Will be in recovery for at least 5:00, or until 258 clients reconnect Aug 8 09:10:50 oss2 kernel: Lustre: 4988:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0018: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 8 09:10:51 oss2 kernel: LDISKFS-fs (dm-9): recovery complete Aug 8 09:10:51 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Aug 8 09:10:51 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Aug 8 09:10:51 oss2 kernel: LDISKFS-fs (dm-9): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:51 oss2 kernel: Lustre: 10858:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0018: 257 recoverable clients remain Aug 8 09:10:51 oss2 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode Aug 8 09:10:51 oss2 kernel: Lustre: 5125:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0004, 258 recoverable clients, 0 delayed clients, last_rcvd 47255779226 Aug 8 09:10:51 oss2 kernel: JBD: barrier-based sync failed on dm-9-8 - disabling barriers Aug 8 09:10:51 oss2 kernel: Lustre: lustre-OST0004: Now serving lustre-OST0004 on /dev/mapper/ost_lustre_4 with recovery enabled Aug 8 09:10:51 oss2 kernel: Lustre: lustre-OST0004: Will be in recovery for at least 5:00, or until 258 clients reconnect Aug 8 09:10:51 oss2 kernel: Lustre: 5125:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0004: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 8 09:10:51 oss2 kernel: LDISKFS-fs warning (device dm-13): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:51 oss2 kernel: Aug 8 09:10:51 oss2 kernel: Lustre: 16666:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 255 recoverable clients remain Aug 8 09:10:51 oss2 kernel: Lustre: 16666:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 7 previous similar messages Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-5): recovery complete Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-25): recovery complete Aug 8 09:10:52 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Aug 8 09:10:52 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Aug 8 09:10:52 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 242 recoverable clients remain Aug 8 09:10:52 oss2 kernel: Lustre: 13212:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 26 previous similar messages Aug 8 09:10:52 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Aug 8 09:10:53 oss2 kernel: LDISKFS-fs (dm-5): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:53 oss2 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode Aug 8 09:10:53 oss2 kernel: LDISKFS-fs (dm-25): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:10:53 oss2 kernel: Lustre: 5429:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0000, 258 recoverable clients, 0 delayed clients, last_rcvd 47255840077 Aug 8 09:10:53 oss2 kernel: JBD: barrier-based sync failed on dm-5-8 - disabling barriers Aug 8 09:10:53 oss2 kernel: Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/mapper/ost_lustre_0 with recovery enabled Aug 8 09:10:53 oss2 kernel: Lustre: lustre-OST0000: Will be in recovery for at least 5:00, or until 258 clients reconnect Aug 8 09:10:53 oss2 kernel: Lustre: 5429:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0000: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 8 09:10:53 oss2 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode Aug 8 09:10:53 oss2 kernel: JBD: barrier-based sync failed on dm-25-8 - disabling barriers Aug 8 09:10:53 oss2 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:53 oss2 kernel: Aug 8 09:10:53 oss2 kernel: LDISKFS-fs warning (device dm-17): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Aug 8 09:10:53 oss2 kernel: Aug 8 09:10:54 oss2 kernel: Lustre: 5182:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0014: 244 recoverable clients remain Aug 8 09:10:54 oss2 kernel: Lustre: 5182:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 69 previous similar messages Aug 8 09:10:58 oss2 kernel: Lustre: 15547:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 175 recoverable clients remain Aug 8 09:10:58 oss2 kernel: Lustre: 15547:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 182 previous similar messages Aug 8 09:11:06 oss2 kernel: Lustre: 25835:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0004: 78 recoverable clients remain Aug 8 09:11:06 oss2 kernel: Lustre: 25835:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 390 previous similar messages Aug 8 09:11:35 oss2 kernel: LustreError: 10836:0:(ldlm_lib.c:944:target_handle_connect()) lustre-OST0004: denying connection for new client 192.168.11.3@o2ib (1b84f639-372e-485b-3d02-733b790a0ab6): 1 clients in recovery for 256s Aug 8 09:11:35 oss2 kernel: LustreError: 10836:0:(ldlm_lib.c:944:target_handle_connect()) Skipped 1 previous similar message Aug 8 09:11:35 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:35 oss2 kernel: LDISKFS-fs (dm-13): recovery complete Aug 8 09:11:35 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Aug 8 09:11:35 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Aug 8 09:11:36 oss2 kernel: LDISKFS-fs (dm-13): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:36 oss2 kernel: LDISKFS-fs (dm-13): mounted filesystem with ordered data mode Aug 8 09:11:36 oss2 kernel: Lustre: 6030:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0008, 258 recoverable clients, 0 delayed clients, last_rcvd 47255868283 Aug 8 09:11:36 oss2 kernel: Lustre: 6030:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Aug 8 09:11:36 oss2 kernel: JBD: barrier-based sync failed on dm-13-8 - disabling barriers Aug 8 09:11:36 oss2 kernel: Lustre: lustre-OST0008: Now serving lustre-OST0008 on /dev/mapper/ost_lustre_8 with recovery enabled Aug 8 09:11:36 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:36 oss2 kernel: Lustre: lustre-OST0008: Will be in recovery for at least 5:00, or until 258 clients reconnect Aug 8 09:11:36 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:36 oss2 kernel: Lustre: 6030:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0008: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 8 09:11:36 oss2 kernel: Lustre: 6030:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Aug 8 09:11:36 oss2 kernel: Lustre: 5167:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0008: 257 recoverable clients remain Aug 8 09:11:36 oss2 kernel: Lustre: 5167:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 340 previous similar messages Aug 8 09:11:36 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-17): recovery complete Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Aug 8 09:11:37 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-17): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-17): mounted filesystem with ordered data mode Aug 8 09:11:37 oss2 kernel: JBD: barrier-based sync failed on dm-17-8 - disabling barriers Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-21): recovery complete Aug 8 09:11:37 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Aug 8 09:11:38 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Aug 8 09:11:38 oss2 kernel: LDISKFS-fs (dm-21): warning: maximal mount count reached, running e2fsck is recommended Aug 8 09:11:38 oss2 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode Aug 8 09:11:38 oss2 kernel: Lustre: 6181:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0010, 258 recoverable clients, 0 delayed clients, last_rcvd 47255879955 Aug 8 09:11:38 oss2 kernel: Lustre: 6181:0:(filter.c:1001:filter_init_server_data()) Skipped 1 previous similar message Aug 8 09:11:38 oss2 kernel: JBD: barrier-based sync failed on dm-21-8 - disabling barriers Aug 8 09:11:38 oss2 kernel: Lustre: lustre-OST0010: Now serving lustre-OST0010 on /dev/mapper/ost_lustre_16 with recovery enabled Aug 8 09:11:38 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:38 oss2 kernel: Lustre: lustre-OST0010: Will be in recovery for at least 5:00, or until 258 clients reconnect Aug 8 09:11:38 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:38 oss2 kernel: Lustre: 6181:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0010: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 8 09:11:38 oss2 kernel: Lustre: 6181:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) Skipped 1 previous similar message Aug 8 09:11:40 oss2 kernel: Lustre: Failing over lustre-OST0018 Aug 8 09:11:40 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 09:11:41 oss2 kernel: Lustre: lustre-OST0014: shutting down for failover; client state will be preserved. Aug 8 09:11:41 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 09:11:41 oss2 kernel: Lustre: OST lustre-OST0014 has stopped. Aug 8 09:11:41 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 09:11:41 oss2 kernel: Lustre: server umount lustre-OST0014 complete Aug 8 09:11:41 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 09:11:41 oss2 kernel: Lustre: lustre-OST0004: shutting down for failover; client state will be preserved. Aug 8 09:11:41 oss2 kernel: Lustre: OST lustre-OST0004 has stopped. Aug 8 09:11:41 oss2 kernel: Lustre: server umount lustre-OST0004 complete Aug 8 09:11:42 oss2 kernel: Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Aug 8 09:11:42 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:42 oss2 kernel: Lustre: OST lustre-OST0000 has stopped. Aug 8 09:11:42 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:42 oss2 kernel: Lustre: server umount lustre-OST0000 complete Aug 8 09:11:42 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 8 09:11:42 oss2 kernel: Lustre: Failing over lustre-OST000c Aug 8 09:11:42 oss2 kernel: Lustre: Skipped 3 previous similar messages Aug 8 09:11:43 oss2 kernel: Lustre: lustre-OST0008: shutting down for failover; client state will be preserved. Aug 8 09:11:43 oss2 kernel: Lustre: OST lustre-OST0008 has stopped. Aug 8 09:11:43 oss2 kernel: Lustre: server umount lustre-OST0008 complete Aug 8 10:34:42 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 10:34:56 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Aug 8 10:35:20 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 10:35:23 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 10:35:40 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 10:35:43 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 10:35:50 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 10:36:04 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Aug 8 10:36:23 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 10:36:26 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 10:38:05 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client 4bcb1673-5d00-756a-cd41-3cc9373f9c27 (at 192.168.2.46@o2ib) in 228 seconds. I think it's dead, and I am evicting it. Aug 8 10:38:05 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 8 10:38:05 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client d2a2d370-35df-80c5-6c71-7dd2e27da3ea (at 192.168.2.116@o2ib) in 218 seconds. I think it's dead, and I am evicting it. Aug 8 10:38:05 oss2 kernel: Lustre: Skipped 77 previous similar messages Aug 8 14:41:06 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 14:43:43 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Aug 8 14:44:08 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 14:44:10 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 14:44:27 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 14:44:30 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 14:44:38 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 14:44:51 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Aug 8 14:44:58 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 14:45:00 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 14:45:18 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 8 Aug 8 14:45:18 oss2 kernel: usb 1-5: configuration #1 chosen from 1 choice Aug 8 14:45:18 oss2 kernel: scsi13 : SCSI emulation for USB Mass Storage devices Aug 8 14:45:18 oss2 kernel: usb-storage: device found at 8 Aug 8 14:45:18 oss2 kernel: usb-storage: waiting for device to settle before scanning Aug 8 14:45:18 oss2 kernel: scsi14 : SCSI emulation for USB Mass Storage devices Aug 8 14:45:18 oss2 kernel: usb-storage: device found at 8 Aug 8 14:45:18 oss2 kernel: usb-storage: waiting for device to settle before scanning Aug 8 14:45:18 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input5 Aug 8 14:45:18 oss2 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Aug 8 14:45:18 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input6 Aug 8 14:45:18 oss2 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Aug 8 14:45:23 oss2 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Aug 8 14:45:23 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Aug 8 14:45:23 oss2 kernel: Vendor: PepperC Model: Virtual Disc 2 Rev: 0.01 Aug 8 14:45:23 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Aug 8 14:45:23 oss2 kernel: sr1: scsi-1 drive Aug 8 14:45:23 oss2 kernel: sr 13:0:0:0: Attached scsi CD-ROM sr1 Aug 8 14:45:23 oss2 kernel: sr 13:0:0:0: Attached scsi generic sg2 type 5 Aug 8 14:45:23 oss2 kernel: sr2: scsi-1 drive Aug 8 14:45:23 oss2 kernel: sr 14:0:0:0: Attached scsi CD-ROM sr2 Aug 8 14:45:23 oss2 kernel: sr 14:0:0:0: Attached scsi generic sg3 type 5 Aug 8 14:45:23 oss2 kernel: usb-storage: device scan complete Aug 8 14:45:23 oss2 kernel: usb-storage: device scan complete Aug 8 15:12:05 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 8 16:44:13 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 16:44:37 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 16:44:44 oss2 kernel: igb: eth0 NIC Link is Down Aug 8 16:45:18 oss2 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Aug 8 18:12:57 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 8 18:12:57 oss2 kernel: LustreError: Skipped 4921 previous similar messages Aug 8 18:12:57 oss2 kernel: LustreError: 13223:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109cc7ed000 x1409790695703374/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344471277 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 18:12:57 oss2 kernel: LustreError: 13223:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4932 previous similar messages Aug 8 18:13:38 oss2 kernel: Lustre: 5172:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 18:13:38 oss2 kernel: Lustre: 5172:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 8 18:20:38 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 18:20:38 oss2 kernel: Lustre: Skipped 1331 previous similar messages Aug 8 18:21:15 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 18:21:15 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 18:21:15 oss2 kernel: LustreError: 16663:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611962050 x1409790695703913/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344471775 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 18:21:15 oss2 kernel: LustreError: 16663:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 18:33:30 oss2 kernel: Lustre: 13293:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 18:33:30 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 18:33:30 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Aug 8 18:33:30 oss2 kernel: Lustre: 13293:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Aug 8 18:58:39 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 18:58:39 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 18:58:39 oss2 kernel: LustreError: 13176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81051fa6d400 x1409790695706493/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344474019 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 18:58:39 oss2 kernel: LustreError: 13176:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 18:59:07 oss2 kernel: Lustre: 10815:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 18:59:07 oss2 kernel: Lustre: 13248:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 18:59:07 oss2 kernel: Lustre: 13248:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 18:59:07 oss2 kernel: Lustre: 10815:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 19:06:38 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 19:06:38 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 19:06:38 oss2 kernel: LustreError: 5156:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103fe865000 x1409790695707017/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344474498 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 19:06:38 oss2 kernel: LustreError: 5156:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 19:07:25 oss2 kernel: Lustre: 24883:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:07:25 oss2 kernel: Lustre: 15519:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:07:25 oss2 kernel: Lustre: 15519:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 8 19:07:25 oss2 kernel: Lustre: 24883:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 8 19:13:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 19:13:49 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 19:13:49 oss2 kernel: LustreError: 15535:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c38ec5000 x1409790695707502/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344474929 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 19:13:49 oss2 kernel: LustreError: 15535:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 19:14:31 oss2 kernel: Lustre: 15541:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:14:31 oss2 kernel: Lustre: 13191:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:14:31 oss2 kernel: Lustre: 13191:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 19:14:31 oss2 kernel: Lustre: 15541:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 19:22:15 oss2 kernel: Lustre: 18409:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:22:15 oss2 kernel: Lustre: 24894:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:22:15 oss2 kernel: Lustre: 24894:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 19:22:15 oss2 kernel: Lustre: 18409:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 19:40:36 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 19:40:36 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 19:40:36 oss2 kernel: LustreError: 10879:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81079b372000 x1409790695714867/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344476536 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 19:40:36 oss2 kernel: LustreError: 10879:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 19:40:52 oss2 kernel: Lustre: 18415:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 19:40:52 oss2 kernel: Lustre: 18415:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 20:05:25 oss2 kernel: Lustre: 5198:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 20:05:25 oss2 kernel: Lustre: 5198:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 20:20:12 oss2 kernel: Lustre: 13199:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 20:20:12 oss2 kernel: Lustre: 13199:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 20:20:12 oss2 kernel: Lustre: 25836:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 20:26:49 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 8 20:26:49 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 20:26:49 oss2 kernel: LustreError: 5167:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81034ee85400 x1409790695718448/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344479309 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 20:26:49 oss2 kernel: LustreError: 5167:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 20:27:06 oss2 kernel: Lustre: 10861:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 20:27:06 oss2 kernel: Lustre: 10861:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 8 20:27:11 oss2 kernel: Lustre: 15532:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 20:27:11 oss2 kernel: Lustre: 15532:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Aug 8 20:48:58 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 20:48:58 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 8 20:49:18 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 20:49:18 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 20:49:18 oss2 kernel: LustreError: 16659:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103dd18d000 x1409790695720111/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344480658 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 20:49:18 oss2 kernel: LustreError: 16659:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:13:01 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 8 21:13:01 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 21:13:01 oss2 kernel: LustreError: 5167:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810921dbd000 x1409790695721774/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344482081 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 21:13:01 oss2 kernel: LustreError: 5167:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:13:33 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 21:13:33 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 8 21:13:33 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 21:13:33 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 21:33:12 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 21:33:12 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 21:33:12 oss2 kernel: LustreError: 18436:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102fb6e3000 x1409790695723079/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344483292 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 21:33:12 oss2 kernel: LustreError: 18436:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:33:40 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 21:33:40 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 21:38:07 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 21:38:07 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 8 21:38:36 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 8 21:38:36 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 21:38:36 oss2 kernel: LustreError: 18423:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103a5369400 x1409790695723464/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344483616 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 21:38:36 oss2 kernel: LustreError: 18423:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:53:19 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 8 21:53:19 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 21:53:19 oss2 kernel: LustreError: 25840:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81033b097c00 x1409790695724465/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344484499 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 21:53:19 oss2 kernel: LustreError: 25840:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:53:33 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 8 21:53:33 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 8 21:56:48 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 8 21:56:48 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 8 21:56:48 oss2 kernel: LustreError: 5257:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c306cac00 x1409790695724706/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344484708 ref 1 fl Interpret:/0/0 rc -19/0 Aug 8 21:56:48 oss2 kernel: LustreError: 5257:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 8 21:57:16 oss2 kernel: Lustre: 15508:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 8 21:57:16 oss2 kernel: Lustre: 15508:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9 previous similar messages Aug 9 00:03:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 9 00:03:02 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 00:03:02 oss2 kernel: LustreError: 18459:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109b9ce5400 x1409790695733311/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344492282 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 00:03:02 oss2 kernel: LustreError: 18459:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 00:03:13 oss2 kernel: Lustre: 13289:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 01:49:15 oss2 kernel: Lustre: 10823:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 01:49:15 oss2 kernel: Lustre: 10823:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Aug 9 01:49:15 oss2 kernel: Lustre: 24896:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from c29dee99-04ac-437e-39e3-57d93e27263b@192.168.2.102@o2ib to 0xffff810c35dc2a00; still busy with 1 active RPCs Aug 9 01:49:15 oss2 kernel: LustreError: 24896:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3df97400 x1409790695884193/t0 o8->c29dee99-04ac-437e-39e3-57d93e27263b@NET_0x50000c0a80266_UUID:0/0 lens 368/264 e 0 to 0 dl 1344498655 ref 1 fl Interpret:/0/0 rc -16/0 Aug 9 01:49:15 oss2 kernel: LustreError: 24896:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 01:49:20 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 01:49:20 oss2 kernel: Lustre: 13240:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 9 01:52:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 9 01:52:45 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 01:52:45 oss2 kernel: LustreError: 15516:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103dd18d800 x1409790695884491/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344498865 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 01:53:12 oss2 kernel: Lustre: 5243:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 01:53:12 oss2 kernel: Lustre: 5243:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 9 02:02:54 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 9 02:02:54 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 9 02:04:12 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 9 02:04:12 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 02:04:12 oss2 kernel: LustreError: 5187:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81043b2f3c00 x1409790695885310/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344499552 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 02:04:12 oss2 kernel: LustreError: 5187:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 02:31:32 oss2 kernel: Lustre: 25839:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 02:31:32 oss2 kernel: Lustre: 13265:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 02:31:32 oss2 kernel: Lustre: 13265:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 9 02:31:32 oss2 kernel: Lustre: 25839:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 9 02:57:35 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 9 02:57:35 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 9 02:57:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0004_UUID' is not available for connect (no target) Aug 9 02:57:45 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 02:57:45 oss2 kernel: LustreError: 5183:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109b4553000 x1409790695888875/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344502765 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 02:57:45 oss2 kernel: LustreError: 5183:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 02:57:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 9 02:57:45 oss2 kernel: LustreError: Skipped 4 previous similar messages Aug 9 03:18:16 oss2 kernel: Lustre: 13199:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 03:18:16 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 03:18:16 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 9 03:18:16 oss2 kernel: Lustre: 13199:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 9 03:45:45 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 9 03:45:45 oss2 kernel: LustreError: Skipped 1 previous similar message Aug 9 03:45:45 oss2 kernel: LustreError: 24885:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103147ecc00 x1409790695892211/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344505645 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 03:45:45 oss2 kernel: LustreError: 24885:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 03:45:45 oss2 kernel: LustreError: 13251:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810176b84400 x1409790695892215/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344505645 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 03:45:45 oss2 kernel: LustreError: 13251:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages Aug 9 03:46:14 oss2 kernel: Lustre: 13259:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 03:46:14 oss2 kernel: Lustre: 13259:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6 previous similar messages Aug 9 03:54:05 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 9 03:54:05 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 9 03:54:05 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 9 03:54:05 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 9 03:54:36 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 9 03:54:36 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 03:54:36 oss2 kernel: LustreError: 13238:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810c0e888400 x1409790695892785/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344506176 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 04:51:20 oss2 kernel: Lustre: 5218:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 04:51:20 oss2 kernel: Lustre: 5218:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 9 09:51:44 oss2 kernel: Lustre: 13289:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 09:51:44 oss2 kernel: Lustre: 13289:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 9 09:54:17 oss2 kernel: Lustre: 15524:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 09:54:17 oss2 kernel: Lustre: 15524:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 10 previous similar messages Aug 9 10:25:53 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 9 10:25:53 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 10:25:53 oss2 kernel: LustreError: 18453:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810611677c50 x1409790695923594/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344529653 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 10:25:53 oss2 kernel: LustreError: 18453:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 10:26:11 oss2 kernel: Lustre: 16662:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 10:26:11 oss2 kernel: Lustre: 16662:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 9 12:01:31 oss2 kernel: Lustre: 10810:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 12:01:31 oss2 kernel: Lustre: 10810:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 9 12:01:31 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 9 12:01:31 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 12:01:31 oss2 kernel: LustreError: 24885:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81036fac5c00 x1409790695931382/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344535391 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 12:01:31 oss2 kernel: LustreError: 24885:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 12:01:32 oss2 kernel: Lustre: 13171:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 12:01:32 oss2 kernel: Lustre: 13171:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 9 12:26:19 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 9 12:26:19 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 9 12:26:19 oss2 kernel: LustreError: 13250:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105229c2800 x1409790695933287/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344536879 ref 1 fl Interpret:/0/0 rc -19/0 Aug 9 12:26:19 oss2 kernel: LustreError: 13250:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 9 12:26:32 oss2 kernel: Lustre: 18475:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 12:26:32 oss2 kernel: Lustre: 16662:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 12:26:32 oss2 kernel: Lustre: 16662:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 9 12:26:32 oss2 kernel: Lustre: 18475:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 9 16:29:14 oss2 kernel: Lustre: 24875:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 16:29:14 oss2 kernel: Lustre: 24875:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 9 21:16:44 oss2 kernel: Lustre: 25836:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 21:16:44 oss2 kernel: Lustre: 25836:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 9 22:42:09 oss2 kernel: Lustre: 5237:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 9 22:42:09 oss2 kernel: Lustre: 5237:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 10 01:16:45 oss2 kernel: Lustre: 13189:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 10 01:16:45 oss2 kernel: Lustre: 13189:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 10 02:19:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 10 02:19:21 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 10 02:19:21 oss2 kernel: LustreError: 5210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810826630400 x1409790696143397/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344586861 ref 1 fl Interpret:/0/0 rc -19/0 Aug 10 02:19:21 oss2 kernel: LustreError: 5210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 10 02:19:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0014_UUID' is not available for connect (no target) Aug 10 02:19:21 oss2 kernel: LustreError: Skipped 1 previous similar message Aug 10 02:19:21 oss2 kernel: LustreError: 15558:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109dd744000 x1409790696143405/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1344586861 ref 1 fl Interpret:/0/0 rc -19/0 Aug 10 02:19:21 oss2 kernel: LustreError: 15558:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Aug 10 02:19:48 oss2 kernel: Lustre: 25840:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 10 02:19:48 oss2 kernel: Lustre: 25840:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 10 08:30:03 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: c29dee99-04ac-437e-39e3-57d93e27263b reconnecting Aug 10 08:30:03 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8 previous similar messages Aug 10 08:33:50 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 10 08:33:50 oss2 kernel: Lustre: Skipped 3 previous similar messages Aug 10 15:43:51 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 249936f5-c39e-f74e-ac1e-7e82dc536bcc (at 192.168.20.2@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 10 15:43:51 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 11 04:12:16 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client 320d110f-b39a-5e31-1b31-306998318258 (at 192.168.2.85@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 11 04:12:16 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 11 21:36:40 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 14:09:27 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 19:10:38 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 19:10:38 oss2 kernel: usb 1-5: device descriptor read/all, error -71 Aug 12 19:10:39 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 20:14:00 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 20:40:15 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 20:41:28 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 20:49:16 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 21:45:54 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 22:35:51 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:03:47 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:09:01 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:16:12 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:19:18 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:20:02 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:20:36 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 12 23:38:18 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 00:03:33 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 00:18:21 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 01:07:47 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 01:13:25 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 01:35:58 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 02:09:50 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 04:50:04 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 04:57:15 oss2 kernel: usb 1-5: reset high speed USB device using ehci_hcd and address 8 Aug 13 04:57:15 oss2 kernel: usb 1-5: failed to restore interface 1 altsetting 0 (error=-71) Aug 13 04:57:15 oss2 kernel: usb 1-5: USB disconnect, address 8 Aug 13 04:57:15 oss2 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 9 Aug 13 04:57:15 oss2 kernel: usb 1-5: configuration #1 chosen from 1 choice Aug 13 04:57:15 oss2 kernel: scsi15 : SCSI emulation for USB Mass Storage devices Aug 13 04:57:15 oss2 kernel: usb-storage: device found at 9 Aug 13 04:57:15 oss2 kernel: usb-storage: waiting for device to settle before scanning Aug 13 04:57:15 oss2 kernel: scsi16 : SCSI emulation for USB Mass Storage devices Aug 13 04:57:15 oss2 kernel: usb-storage: device found at 9 Aug 13 04:57:15 oss2 kernel: usb-storage: waiting for device to settle before scanning Aug 13 04:57:15 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input7 Aug 13 04:57:15 oss2 kernel: input: USB HID v1.01 Mouse [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Aug 13 04:57:16 oss2 kernel: input: Peppercon AG Multidevice as /class/input/input8 Aug 13 04:57:16 oss2 kernel: input: USB HID v1.01 Keyboard [Peppercon AG Multidevice] on usb-0000:00:1a.7-5 Aug 13 04:57:20 oss2 kernel: Vendor: PepperC Model: Virtual Disc 2 Rev: 0.01 Aug 13 04:57:20 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Aug 13 04:57:20 oss2 kernel: Vendor: PepperC Model: Virtual Disc 1 Rev: 0.01 Aug 13 04:57:20 oss2 kernel: Type: CD-ROM ANSI SCSI revision: 03 Aug 13 04:57:21 oss2 kernel: sr2: scsi-1 drive Aug 13 04:57:21 oss2 kernel: sr 15:0:0:0: Attached scsi CD-ROM sr2 Aug 13 04:57:21 oss2 kernel: sr 15:0:0:0: Attached scsi generic sg2 type 5 Aug 13 04:57:21 oss2 kernel: sr1: scsi-1 drive Aug 13 04:57:21 oss2 kernel: sr 16:0:0:0: Attached scsi CD-ROM sr1 Aug 13 04:57:21 oss2 kernel: sr 16:0:0:0: Attached scsi generic sg3 type 5 Aug 13 04:57:21 oss2 kernel: usb-storage: device scan complete Aug 13 04:57:21 oss2 kernel: usb-storage: device scan complete Aug 20 16:57:44 oss2 kernel: Lustre: 13259:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 31b023c7-59de-a490-0471-b6b9146d82bb reconnecting Aug 21 10:36:01 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client 8572804e-132e-7030-080e-7862388f8010 (at 192.168.2.184@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 21 10:36:01 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 21 10:36:01 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client 8572804e-132e-7030-080e-7862388f8010 (at 192.168.2.184@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 21 10:37:18 oss2 kernel: LustreError: 18440:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0001_UUID: lvbo_init failed for resource 4325648: rc -2 Aug 21 10:46:17 oss2 kernel: LustreError: 25853:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0011_UUID: lvbo_init failed for resource 4325519: rc -2 Aug 21 19:50:29 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client c11d9ca3-0f44-34d6-ecf3-af9489ea99e2 (at 192.168.11.19@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 21 19:50:29 oss2 kernel: Lustre: Skipped 4 previous similar messages Aug 21 22:09:24 oss2 kernel: LustreError: 5222:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0005_UUID: lvbo_init failed for resource 4359126: rc -2 Aug 22 11:03:17 oss2 kernel: Lustre: 13224:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0005: 6cf0bb0c-d97e-c225-8593-ba90311f1627 reconnecting Aug 22 11:03:17 oss2 kernel: Lustre: 13224:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages Aug 22 11:03:17 oss2 kernel: Lustre: 24877:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 6cf0bb0c-d97e-c225-8593-ba90311f1627@192.168.3.14@o2ib to 0xffff810a0e41c400; still busy with 1 active RPCs Aug 22 11:03:17 oss2 kernel: LustreError: 24877:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810611895450 x1401514823136021/t0 o8->6cf0bb0c-d97e-c225-8593-ba90311f1627@NET_0x50000c0a8030e_UUID:0/0 lens 368/264 e 0 to 0 dl 1345655097 ref 1 fl Interpret:/0/0 rc -16/0 Aug 22 11:03:17 oss2 kernel: LustreError: 24877:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Aug 22 11:03:21 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 6cf0bb0c-d97e-c225-8593-ba90311f1627 reconnecting Aug 22 11:03:21 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 22 14:09:56 oss2 kernel: LustreError: 24889:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0001_UUID: lvbo_init failed for resource 4371419: rc -2 Aug 22 14:47:20 oss2 kernel: LustreError: 18448:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0015_UUID: lvbo_init failed for resource 4373158: rc -2 Aug 23 10:44:12 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 123s due to heavy IO load Aug 23 10:47:39 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 177s due to heavy IO load Aug 23 11:04:56 oss2 kernel: Lustre: Service thread pid 4432 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 11:04:56 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 11:04:56 oss2 kernel: Pid: 4432, comm: ll_ost_io_341 Aug 23 11:04:56 oss2 kernel: Aug 23 11:04:56 oss2 kernel: Call Trace: Aug 23 11:04:56 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 11:04:56 oss2 kernel: [] try_to_wake_up+0x472/0x484 Aug 23 11:04:56 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:04:56 oss2 kernel: [] __down_trylock+0x13/0x4e Aug 23 11:04:56 oss2 kernel: [] __down_failed_trylock+0x35/0x3a Aug 23 11:04:56 oss2 kernel: [] __down_failed+0x35/0x3a Aug 23 11:04:56 oss2 kernel: [] acpi_cpufreq_target+0x0/0x3f8 [acpi_cpufreq] Aug 23 11:04:56 oss2 kernel: [] .text.lock.ldlm_pool+0x55/0x71 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 11:04:56 oss2 kernel: [] __up_read+0x19/0x7f Aug 23 11:04:56 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 11:04:56 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 11:04:56 oss2 kernel: [] d_rehash_cond+0x2c/0x46 Aug 23 11:04:56 oss2 kernel: [] __rmqueue+0x91/0xc6 Aug 23 11:04:56 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 11:04:56 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 11:04:56 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 11:04:56 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] filter_preprw+0x14da/0x1e00 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Aug 23 11:04:56 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 11:04:56 oss2 kernel: [] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Aug 23 11:04:56 oss2 kernel: [] ost_brw_write+0xf9c/0x2480 [ost] Aug 23 11:04:56 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 11:04:56 oss2 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] filter_truncate+0x1d9/0x270 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Aug 23 11:04:56 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 11:04:56 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 11:04:56 oss2 kernel: Aug 23 11:04:56 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345741496.4432 Aug 23 11:04:56 oss2 kernel: Lustre: Service thread pid 7881 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 11:04:56 oss2 kernel: Pid: 7881, comm: ll_ost_io_231 Aug 23 11:04:56 oss2 kernel: Aug 23 11:04:56 oss2 kernel: Call Trace: Aug 23 11:04:56 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 11:04:56 oss2 kernel: [] try_to_wake_up+0x472/0x484 Aug 23 11:04:56 oss2 kernel: [] __down+0xc3/0xd8 Aug 23 11:04:56 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 11:04:56 oss2 kernel: [] __down_trylock+0x44/0x4e Aug 23 11:04:56 oss2 kernel: [] __down_failed_trylock+0x35/0x3a Aug 23 11:04:56 oss2 kernel: [] acpi_cpufreq_target+0x0/0x3f8 [acpi_cpufreq] Aug 23 11:04:56 oss2 kernel: [] __down_failed+0x35/0x3a Aug 23 11:04:56 oss2 kernel: [] acpi_cpufreq_target+0x0/0x3f8 [acpi_cpufreq] Aug 23 11:04:56 oss2 kernel: [] .text.lock.ldlm_resource+0x73/0x87 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] .text.lock.ldlm_pool+0x55/0x71 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 11:04:56 oss2 kernel: [] __up_read+0x19/0x7f Aug 23 11:04:56 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 11:04:56 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 11:04:56 oss2 kernel: [] __d_lookup+0xb0/0xff Aug 23 11:04:56 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 11:04:56 oss2 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 11:04:56 oss2 kernel: [] kiblnd_post_tx_locked+0x566/0x730 [ko2iblnd] Aug 23 11:04:56 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 11:04:56 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] filter_preprw_read+0x601/0xd30 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Aug 23 11:04:56 oss2 kernel: [] lookup_one_len+0x53/0x61 Aug 23 11:04:56 oss2 kernel: [] filter_fid2dentry+0x512/0x740 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] filter_preprw+0x1d9c/0x1e00 [obdfilter] Aug 23 11:04:56 oss2 kernel: [] lnet_ni_send+0x96/0xe0 [lnet] Aug 23 11:04:56 oss2 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Aug 23 11:04:56 oss2 kernel: [] lnet_setpayloadbuffer+0xa6/0x120 [lnet] Aug 23 11:04:56 oss2 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 11:04:56 oss2 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] ost_brw_read+0xb33/0x1a70 [ost] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Aug 23 11:04:56 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 11:04:56 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 11:04:56 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 11:04:56 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 11:04:56 oss2 kernel: Aug 23 11:04:56 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345741496.7881 Aug 23 11:15:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 23 11:15:02 oss2 kernel: LustreError: Skipped 4 previous similar messages Aug 23 11:15:02 oss2 kernel: LustreError: 5213:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81066709ac00 x1409778622820362/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742202 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:02 oss2 kernel: LustreError: 5210:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810613c8ec50 x1409779447198313/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742202 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:02 oss2 kernel: LustreError: 5210:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Aug 23 11:15:02 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 23 11:15:02 oss2 kernel: LustreError: Skipped 5 previous similar messages Aug 23 11:15:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 23 11:15:03 oss2 kernel: LustreError: Skipped 14 previous similar messages Aug 23 11:15:03 oss2 kernel: LustreError: 13255:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061990d800 x1410983156405020/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742203 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:03 oss2 kernel: LustreError: 13255:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 17 previous similar messages Aug 23 11:15:03 oss2 kernel: Lustre: 4381:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-207), not sending early reply Aug 23 11:15:03 oss2 kernel: req@ffff810ad8023000 x1409777946137095/t0 o4->1992e104-b81d-3d88-0be0-5ba2904aa681@NET_0x50000c0a80213_UUID:0/0 lens 448/416 e 5 to 0 dl 1345742108 ref 2 fl Interpret:/0/0 rc 0/0 Aug 23 11:15:04 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 23 11:15:04 oss2 kernel: LustreError: Skipped 23 previous similar messages Aug 23 11:15:04 oss2 kernel: LustreError: 25859:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810625cb1800 x1409778923943812/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742204 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:04 oss2 kernel: LustreError: 25859:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 23 previous similar messages Aug 23 11:15:06 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 23 11:15:06 oss2 kernel: LustreError: Skipped 56 previous similar messages Aug 23 11:15:06 oss2 kernel: LustreError: 5204:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810bc7ca2000 x1409778227283073/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742206 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:06 oss2 kernel: LustreError: 5204:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 56 previous similar messages Aug 23 11:15:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0010_UUID' is not available for connect (no target) Aug 23 11:15:10 oss2 kernel: LustreError: Skipped 118 previous similar messages Aug 23 11:15:10 oss2 kernel: LustreError: 10850:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105aa6d8000 x1409778915603193/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742210 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:10 oss2 kernel: LustreError: 10850:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 118 previous similar messages Aug 23 11:15:18 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 23 11:15:18 oss2 kernel: LustreError: Skipped 547 previous similar messages Aug 23 11:15:18 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810618dd2800 x1409778643842155/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742218 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:18 oss2 kernel: LustreError: 13262:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 547 previous similar messages Aug 23 11:15:34 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 23 11:15:34 oss2 kernel: LustreError: Skipped 765 previous similar messages Aug 23 11:15:34 oss2 kernel: LustreError: 5247:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8107f569f000 x1406028867363856/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345742234 ref 1 fl Interpret:/0/0 rc -19/0 Aug 23 11:15:34 oss2 kernel: LustreError: 5247:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 765 previous similar messages Aug 23 11:16:59 oss2 kernel: Lustre: 18444:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:16:59 oss2 kernel: Lustre: 18444:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:16:59 oss2 kernel: LustreError: 18444:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8101713ac000 x1409777946150609/t0 o8->1992e104-b81d-3d88-0be0-5ba2904aa681@NET_0x50000c0a80213_UUID:0/0 lens 368/264 e 0 to 0 dl 1345742319 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 11:16:59 oss2 kernel: LustreError: 18444:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 141 previous similar messages Aug 23 11:17:01 oss2 kernel: Lustre: 10876:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 4a3e47a6-b399-f11c-1c3a-b3be2792c532 reconnecting Aug 23 11:17:01 oss2 kernel: Lustre: 10876:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 4a3e47a6-b399-f11c-1c3a-b3be2792c532@192.168.2.174@o2ib to 0xffff810818fdfa00; still busy with 1 active RPCs Aug 23 11:17:06 oss2 kernel: Lustre: 18452:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:17:06 oss2 kernel: Lustre: 18452:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:17:08 oss2 kernel: Lustre: 10881:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 4a3e47a6-b399-f11c-1c3a-b3be2792c532 reconnecting Aug 23 11:17:08 oss2 kernel: Lustre: 10881:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 4a3e47a6-b399-f11c-1c3a-b3be2792c532@192.168.2.174@o2ib to 0xffff810818fdfa00; still busy with 1 active RPCs Aug 23 11:17:13 oss2 kernel: Lustre: 18469:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:17:13 oss2 kernel: Lustre: 18469:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:17:20 oss2 kernel: Lustre: 18423:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:17:20 oss2 kernel: Lustre: 18423:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Aug 23 11:17:20 oss2 kernel: Lustre: 18423:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:17:20 oss2 kernel: Lustre: 18423:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message Aug 23 11:17:29 oss2 kernel: Lustre: 25854:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 4a3e47a6-b399-f11c-1c3a-b3be2792c532 reconnecting Aug 23 11:17:29 oss2 kernel: Lustre: 25854:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 23 11:17:29 oss2 kernel: Lustre: 25854:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 4a3e47a6-b399-f11c-1c3a-b3be2792c532@192.168.2.174@o2ib to 0xffff810818fdfa00; still busy with 1 active RPCs Aug 23 11:17:29 oss2 kernel: Lustre: 25854:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages Aug 23 11:17:48 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:17:48 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 23 11:17:48 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:17:48 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 4 previous similar messages Aug 23 11:18:04 oss2 kernel: LustreError: 18431:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8108940e1800 x1409779437459237/t0 o8->4a3e47a6-b399-f11c-1c3a-b3be2792c532@NET_0x50000c0a802ae_UUID:0/0 lens 368/264 e 0 to 0 dl 1345742384 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 11:18:04 oss2 kernel: LustreError: 18431:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18 previous similar messages Aug 23 11:18:23 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 1992e104-b81d-3d88-0be0-5ba2904aa681 reconnecting Aug 23 11:18:23 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9 previous similar messages Aug 23 11:18:23 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 1992e104-b81d-3d88-0be0-5ba2904aa681@192.168.2.19@o2ib to 0xffff810604900a00; still busy with 1 active RPCs Aug 23 11:18:23 oss2 kernel: Lustre: 13235:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 9 previous similar messages Aug 23 11:19:28 oss2 kernel: Lustre: 5199:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST000d: 4a3e47a6-b399-f11c-1c3a-b3be2792c532 reconnecting Aug 23 11:19:28 oss2 kernel: Lustre: 5199:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 18 previous similar messages Aug 23 11:19:28 oss2 kernel: Lustre: 5199:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST000d: refuse reconnection from 4a3e47a6-b399-f11c-1c3a-b3be2792c532@192.168.2.174@o2ib to 0xffff810818fdfa00; still busy with 1 active RPCs Aug 23 11:19:28 oss2 kernel: Lustre: 5199:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 18 previous similar messages Aug 23 11:20:08 oss2 kernel: Lustre: lustre-OST000d: slow start_page_read 1111s due to heavy IO load Aug 23 11:20:08 oss2 kernel: LustreError: 7881:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107 req@ffff8100709eac50 x1409779437449253/t0 o3->4a3e47a6-b399-f11c-1c3a-b3be2792c532@NET_0x50000c0a802ae_UUID:0/0 lens 448/400 e 5 to 0 dl 1345742108 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 11:20:08 oss2 kernel: Lustre: 7881:0:(ost_handler.c:887:ost_brw_read()) lustre-OST000d: ignoring bulk IO comm error with 4a3e47a6-b399-f11c-1c3a-b3be2792c532@NET_0x50000c0a802ae_UUID id 12345-192.168.2.174@o2ib - client will retry Aug 23 11:20:08 oss2 kernel: Lustre: 7881:0:(ost_handler.c:887:ost_brw_read()) Skipped 1 previous similar message Aug 23 11:20:08 oss2 kernel: Lustre: 7881:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1409779437449253 took longer than estimated (812+300s); client may timeout. req@ffff8100709eac50 x1409779437449253/t0 o3->4a3e47a6-b399-f11c-1c3a-b3be2792c532@NET_0x50000c0a802ae_UUID:0/0 lens 448/400 e 5 to 0 dl 1345742108 ref 1 fl Complete:/0/0 rc 0/0 Aug 23 11:20:08 oss2 kernel: Lustre: Service thread pid 7881 completed after 1111.79s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 11:20:08 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 23 11:20:08 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 1112s due to heavy IO load Aug 23 11:20:08 oss2 kernel: LustreError: 4432:0:(ost_handler.c:1078:ost_brw_write()) @@@ ptlrpc_bulk_get failed: rc -107 req@ffff810ad8023000 x1409777946137095/t0 o4->1992e104-b81d-3d88-0be0-5ba2904aa681@NET_0x50000c0a80213_UUID:0/0 lens 448/416 e 5 to 0 dl 1345742108 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 11:20:08 oss2 kernel: Lustre: 4432:0:(ost_handler.c:1224:ost_brw_write()) lustre-OST0001: ignoring bulk IO comm error with 1992e104-b81d-3d88-0be0-5ba2904aa681@NET_0x50000c0a80213_UUID id 12345-192.168.2.19@o2ib - client will retry Aug 23 11:20:08 oss2 kernel: Lustre: 4432:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1409777946137095 took longer than estimated (812+300s); client may timeout. req@ffff810ad8023000 x1409777946137095/t0 o4->1992e104-b81d-3d88-0be0-5ba2904aa681@NET_0x50000c0a80213_UUID:0/0 lens 448/416 e 5 to 0 dl 1345742108 ref 1 fl Complete:/0/0 rc 0/0 Aug 23 11:20:08 oss2 kernel: Lustre: Service thread pid 4432 completed after 1112.45s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 11:48:21 oss2 kernel: Lustre: Service thread pid 13371 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 11:48:21 oss2 kernel: Pid: 13371, comm: ll_ost_io_72 Aug 23 11:48:21 oss2 kernel: Aug 23 11:48:21 oss2 kernel: Call Trace: Aug 23 11:48:21 oss2 kernel: [] try_to_wake_up+0x472/0x484 Aug 23 11:48:21 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:48:21 oss2 kernel: [] __down+0xc3/0xd8 Aug 23 11:48:21 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 11:48:21 oss2 kernel: [] __down_failed+0x35/0x3a Aug 23 11:48:21 oss2 kernel: [] .text.lock.ldlm_resource+0x73/0x87 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] .text.lock.ldlm_pool+0x37/0x71 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 11:48:21 oss2 kernel: [] __up_read+0x19/0x7f Aug 23 11:48:21 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 11:48:21 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 11:48:21 oss2 kernel: [] __rmqueue+0x91/0xc6 Aug 23 11:48:21 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 11:48:21 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 11:48:21 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 11:48:21 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 11:48:21 oss2 kernel: [] filter_preprw+0x14da/0x1e00 [obdfilter] Aug 23 11:48:21 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 11:48:21 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 11:48:21 oss2 kernel: [] ost_brw_write+0xf9c/0x2480 [ost] Aug 23 11:48:21 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 11:48:21 oss2 kernel: [] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 11:48:21 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Aug 23 11:48:21 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 11:48:21 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 11:48:21 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 11:48:21 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 11:48:21 oss2 kernel: Aug 23 11:48:21 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345744101.13371 Aug 23 11:48:37 oss2 kernel: Lustre: Service thread pid 13348 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 11:48:37 oss2 kernel: Pid: 13348, comm: ll_ost_io_49 Aug 23 11:48:37 oss2 kernel: Aug 23 11:48:37 oss2 kernel: Call Trace: Aug 23 11:48:37 oss2 kernel: [] __sched_text_start+0x497/0xbce Aug 23 11:48:37 oss2 kernel: [] try_to_wake_up+0x472/0x484 Aug 23 11:48:37 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:48:37 oss2 kernel: [] __down+0xc3/0xd8 Aug 23 11:48:37 oss2 kernel: [] __down_failed+0x35/0x3a Aug 23 11:48:37 oss2 kernel: [] acpi_cpufreq_target+0x0/0x3f8 [acpi_cpufreq] Aug 23 11:48:37 oss2 kernel: [] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] .text.lock.ldlm_pool+0x55/0x71 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 11:48:37 oss2 kernel: [] __up_read+0x19/0x7f Aug 23 11:48:37 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 11:48:37 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 11:48:37 oss2 kernel: [] __rmqueue+0x44/0xc6 Aug 23 11:48:37 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 11:48:37 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 11:48:37 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 11:48:37 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 11:48:37 oss2 kernel: [] filter_preprw+0x14da/0x1e00 [obdfilter] Aug 23 11:48:37 oss2 kernel: [] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Aug 23 11:48:37 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 11:48:37 oss2 kernel: [] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Aug 23 11:48:37 oss2 kernel: [] ost_brw_write+0xf9c/0x2480 [ost] Aug 23 11:48:37 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 11:48:37 oss2 kernel: [] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Aug 23 11:48:37 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 11:48:37 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 11:48:37 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 11:48:37 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 11:48:37 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 11:48:37 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 11:48:37 oss2 kernel: Aug 23 11:48:37 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345744117.13348 Aug 23 11:49:26 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 249s due to heavy IO load Aug 23 11:49:26 oss2 kernel: Lustre: Service thread pid 13348 completed after 249.39s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 11:49:26 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 265s due to heavy IO load Aug 23 11:49:26 oss2 kernel: Lustre: Service thread pid 13371 completed after 265.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 12:50:48 oss2 kernel: Lustre: lustre-OST0015: slow start_page_read 82s due to heavy IO load Aug 23 12:52:29 oss2 kernel: Lustre: Service thread pid 7842 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 12:52:29 oss2 kernel: Pid: 7842, comm: ll_ost_io_192 Aug 23 12:52:29 oss2 kernel: Aug 23 12:52:29 oss2 kernel: Call Trace: Aug 23 12:52:29 oss2 kernel: [] lcw_cb+0x0/0x460 [libcfs] Aug 23 12:52:29 oss2 kernel: [] libcfs_debug_dumpstack+0x51/0x60 [libcfs] Aug 23 12:52:29 oss2 kernel: [] lcw_cb+0x33c/0x460 [libcfs] Aug 23 12:52:29 oss2 kernel: [] run_timer_softirq+0x193/0x241 Aug 23 12:52:29 oss2 kernel: [] __do_softirq+0x89/0x133 Aug 23 12:52:29 oss2 kernel: [] call_softirq+0x1c/0x28 Aug 23 12:52:29 oss2 kernel: [] do_softirq+0x2c/0x7d Aug 23 12:52:29 oss2 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 23 12:52:29 oss2 kernel: [] acpi_cpufreq_target+0x0/0x3f8 [acpi_cpufreq] Aug 23 12:52:29 oss2 kernel: [] lprocfs_counter_add+0x5b/0x100 [lvfs] Aug 23 12:52:29 oss2 kernel: [] ldlm_pool_shrink+0x50/0xf0 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ldlm_pools_shrink+0x29f/0x2f0 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 12:52:29 oss2 kernel: [] __up_read+0x19/0x7f Aug 23 12:52:29 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 12:52:29 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 12:52:29 oss2 kernel: [] __d_lookup+0xb0/0xff Aug 23 12:52:29 oss2 kernel: [] __rmqueue+0x44/0xc6 Aug 23 12:52:29 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 12:52:29 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 12:52:29 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 12:52:29 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 12:52:29 oss2 kernel: [] filter_preprw_read+0x601/0xd30 [obdfilter] Aug 23 12:52:29 oss2 kernel: [] lnet_send+0x974/0x9d0 [lnet] Aug 23 12:52:29 oss2 kernel: [] ptl_send_buf+0x3f3/0x5b0 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Aug 23 12:52:29 oss2 kernel: [] filter_preprw+0x1d9c/0x1e00 [obdfilter] Aug 23 12:52:29 oss2 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ost_brw_read+0xb33/0x1a70 [ost] Aug 23 12:52:29 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 12:52:29 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Aug 23 12:52:29 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 12:52:29 oss2 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 12:52:29 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 12:52:29 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 12:52:29 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 12:52:29 oss2 kernel: Aug 23 12:52:29 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345747949.7842 Aug 23 12:54:08 oss2 kernel: Lustre: Service thread pid 7855 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 12:54:08 oss2 kernel: Pid: 7855, comm: ll_ost_io_205 Aug 23 12:54:08 oss2 kernel: Aug 23 12:54:08 oss2 kernel: Call Trace: Aug 23 12:54:08 oss2 kernel: [] try_to_wake_up+0x27/0x484 Aug 23 12:54:08 oss2 kernel: [] __down+0xc3/0xd8 Aug 23 12:54:08 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 12:54:08 oss2 kernel: [] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] ldlm_pools_shrink+0x147/0x2f0 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] __down_read+0x12/0x92 Aug 23 12:54:08 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 23 12:54:08 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 23 12:54:08 oss2 kernel: [] __d_lookup+0xb0/0xff Aug 23 12:54:08 oss2 kernel: [] __rmqueue+0x44/0xc6 Aug 23 12:54:08 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 23 12:54:08 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 23 12:54:08 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 23 12:54:08 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 23 12:54:08 oss2 kernel: [] filter_preprw_read+0x601/0xd30 [obdfilter] Aug 23 12:54:08 oss2 kernel: [] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Aug 23 12:54:08 oss2 kernel: [] qctxt_adjust_qunit+0xce/0x340 [lquota] Aug 23 12:54:08 oss2 kernel: [] filter_preprw+0x1d9c/0x1e00 [obdfilter] Aug 23 12:54:08 oss2 kernel: [] lnet_ni_send+0x96/0xe0 [lnet] Aug 23 12:54:08 oss2 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Aug 23 12:54:08 oss2 kernel: [] pop_ctxt+0x290/0x370 [lvfs] Aug 23 12:54:08 oss2 kernel: [] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] LNetMDBind+0x301/0x450 [lnet] Aug 23 12:54:08 oss2 kernel: [] ost_brw_read+0xb33/0x1a70 [ost] Aug 23 12:54:08 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] ost_handle+0x2e73/0x55b0 [ost] Aug 23 12:54:08 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 12:54:08 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 12:54:08 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 12:54:08 oss2 kernel: Aug 23 12:54:08 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345748048.7855 Aug 23 13:06:20 oss2 kernel: Lustre: 4422:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-431), not sending early reply Aug 23 13:06:20 oss2 kernel: req@ffff810a27227400 x1409790232142161/t0 o3->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 448/400 e 5 to 0 dl 1345748785 ref 2 fl Interpret:/0/0 rc 0/0 Aug 23 13:06:20 oss2 kernel: Lustre: 4422:0:(service.c:808:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Aug 23 13:07:47 oss2 kernel: Lustre: 4466:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-420), not sending early reply Aug 23 13:07:47 oss2 kernel: req@ffff8107ad101800 x1409779559983755/t0 o3->2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID:0/0 lens 448/400 e 3 to 0 dl 1345748872 ref 2 fl Interpret:/0/0 rc 0/0 Aug 23 13:08:47 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:08:47 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 12 previous similar messages Aug 23 13:08:47 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:08:47 oss2 kernel: Lustre: 18445:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 10 previous similar messages Aug 23 13:08:47 oss2 kernel: LustreError: 18445:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b65ebf800 x1409790232174891/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749027 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:08:47 oss2 kernel: LustreError: 18445:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 34 previous similar messages Aug 23 13:09:08 oss2 kernel: Lustre: 5171:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:09:08 oss2 kernel: Lustre: 5171:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 23 13:09:08 oss2 kernel: Lustre: 5171:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:09:08 oss2 kernel: Lustre: 5171:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages Aug 23 13:09:08 oss2 kernel: LustreError: 5171:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b9dbb2800 x1409790232175363/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749048 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:09:08 oss2 kernel: LustreError: 5171:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Aug 23 13:09:43 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:09:43 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages Aug 23 13:09:43 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:09:43 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 4 previous similar messages Aug 23 13:09:43 oss2 kernel: LustreError: 13180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8103ab30c400 x1409790232175772/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749083 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:09:43 oss2 kernel: LustreError: 13180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Aug 23 13:10:47 oss2 kernel: Lustre: 18471:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 2b94f6e2-6ed0-1bec-352e-09fcaba969a6 reconnecting Aug 23 13:10:47 oss2 kernel: Lustre: 18471:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 14 previous similar messages Aug 23 13:10:47 oss2 kernel: Lustre: 18471:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 2b94f6e2-6ed0-1bec-352e-09fcaba969a6@192.168.2.227@o2ib to 0xffff81034ae4ce00; still busy with 1 active RPCs Aug 23 13:10:47 oss2 kernel: Lustre: 18471:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 14 previous similar messages Aug 23 13:10:47 oss2 kernel: LustreError: 18471:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81069aaf6000 x1409779559999809/t0 o8->2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749147 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:10:47 oss2 kernel: LustreError: 18471:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 14 previous similar messages Aug 23 13:12:59 oss2 kernel: Lustre: 5185:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:12:59 oss2 kernel: Lustre: 5185:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 36 previous similar messages Aug 23 13:12:59 oss2 kernel: Lustre: 5185:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:12:59 oss2 kernel: Lustre: 5185:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 36 previous similar messages Aug 23 13:12:59 oss2 kernel: LustreError: 5185:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810a3bd2f000 x1409790232177287/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749279 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:12:59 oss2 kernel: LustreError: 5185:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 36 previous similar messages Aug 23 13:17:18 oss2 kernel: Lustre: 10854:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:17:18 oss2 kernel: Lustre: 10854:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 73 previous similar messages Aug 23 13:17:18 oss2 kernel: Lustre: 10854:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:17:18 oss2 kernel: Lustre: 10854:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 73 previous similar messages Aug 23 13:17:18 oss2 kernel: LustreError: 10854:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3d41c400 x1409790232178691/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345749538 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:17:18 oss2 kernel: LustreError: 10854:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 73 previous similar messages Aug 23 13:25:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: 2b94f6e2-6ed0-1bec-352e-09fcaba969a6 reconnecting Aug 23 13:25:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 146 previous similar messages Aug 23 13:25:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0001: refuse reconnection from 2b94f6e2-6ed0-1bec-352e-09fcaba969a6@192.168.2.227@o2ib to 0xffff81034ae4ce00; still busy with 1 active RPCs Aug 23 13:25:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 146 previous similar messages Aug 23 13:25:50 oss2 kernel: LustreError: 5191:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3f401c00 x1409779560000980/t0 o8->2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID:0/0 lens 368/264 e 0 to 0 dl 1345750050 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:25:50 oss2 kernel: LustreError: 5191:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 146 previous similar messages Aug 23 13:35:51 oss2 kernel: Lustre: 5196:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:35:51 oss2 kernel: Lustre: 5196:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 170 previous similar messages Aug 23 13:35:51 oss2 kernel: Lustre: 5196:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:35:51 oss2 kernel: Lustre: 5196:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 170 previous similar messages Aug 23 13:35:51 oss2 kernel: LustreError: 5196:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3ffd3c00 x1409790232182087/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345750651 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:35:51 oss2 kernel: LustreError: 5196:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 170 previous similar messages Aug 23 13:45:53 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:45:53 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 13:45:53 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:45:53 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 13:45:53 oss2 kernel: LustreError: 18411:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106195a3000 x1409790232183918/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345751253 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:45:53 oss2 kernel: LustreError: 18411:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 13:55:55 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 13:55:55 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 13:55:55 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 13:55:55 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 13:55:55 oss2 kernel: LustreError: 5155:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c1ad1e400 x1409790232185770/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345751855 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 13:55:55 oss2 kernel: LustreError: 5155:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:05:57 oss2 kernel: Lustre: 5168:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:05:57 oss2 kernel: Lustre: 5168:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:05:57 oss2 kernel: Lustre: 5168:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:05:57 oss2 kernel: Lustre: 5168:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:05:57 oss2 kernel: LustreError: 5168:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8105c7954000 x1409790232187612/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345752457 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:05:57 oss2 kernel: LustreError: 5168:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:15:59 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:15:59 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:15:59 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:15:59 oss2 kernel: Lustre: 15514:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:15:59 oss2 kernel: LustreError: 15514:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b9dbb2400 x1409790232189424/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345753059 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:15:59 oss2 kernel: LustreError: 15514:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:26:01 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:26:01 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:26:01 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:26:01 oss2 kernel: Lustre: 5155:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:26:01 oss2 kernel: LustreError: 5155:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81021fdbc800 x1409790232190678/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345753661 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:26:01 oss2 kernel: LustreError: 5155:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:36:03 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:36:03 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:36:03 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:36:03 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:36:03 oss2 kernel: LustreError: 24892:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8104254d7400 x1409790232192514/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345754263 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:36:03 oss2 kernel: LustreError: 24892:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:46:05 oss2 kernel: Lustre: 15560:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:46:05 oss2 kernel: Lustre: 15560:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:46:05 oss2 kernel: Lustre: 15560:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:46:05 oss2 kernel: Lustre: 15560:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:46:05 oss2 kernel: LustreError: 15560:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b2ee2d000 x1409790232194320/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345754865 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:46:05 oss2 kernel: LustreError: 15560:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 14:56:07 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 14:56:07 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 14:56:07 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 14:56:07 oss2 kernel: Lustre: 5245:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 14:56:07 oss2 kernel: LustreError: 5245:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c37632c00 x1409790232196153/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345755467 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 14:56:07 oss2 kernel: LustreError: 5245:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:06:09 oss2 kernel: Lustre: 10843:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 15:06:09 oss2 kernel: Lustre: 10843:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 15:06:09 oss2 kernel: Lustre: 10843:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 15:06:09 oss2 kernel: Lustre: 10843:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 15:06:09 oss2 kernel: LustreError: 10843:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81061f39bc00 x1409790232197967/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345756069 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 15:06:09 oss2 kernel: LustreError: 10843:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:16:11 oss2 kernel: Lustre: 10875:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 15:16:11 oss2 kernel: Lustre: 10875:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 15:16:11 oss2 kernel: Lustre: 10875:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 15:16:11 oss2 kernel: Lustre: 10875:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 15:16:11 oss2 kernel: LustreError: 10875:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106072fb800 x1409790232199790/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345756671 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 15:16:11 oss2 kernel: LustreError: 10875:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:26:13 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 15:26:13 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 15:26:13 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 15:26:13 oss2 kernel: Lustre: 5226:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 15:26:13 oss2 kernel: LustreError: 5226:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810ab02e9c00 x1409790232201615/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345757273 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 15:26:13 oss2 kernel: LustreError: 5226:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:36:15 oss2 kernel: Lustre: 13257:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 15:36:15 oss2 kernel: Lustre: 13257:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 15:36:15 oss2 kernel: Lustre: 13257:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 15:36:15 oss2 kernel: Lustre: 13257:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 15:36:15 oss2 kernel: LustreError: 13257:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81033b097800 x1409790232203429/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345757875 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 15:36:15 oss2 kernel: LustreError: 13257:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:46:17 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0015: 78575d7a-52f5-c0fb-4a05-d9810d84195a reconnecting Aug 23 15:46:17 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 171 previous similar messages Aug 23 15:46:17 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0015: refuse reconnection from 78575d7a-52f5-c0fb-4a05-d9810d84195a@192.168.2.86@o2ib to 0xffff810807766800; still busy with 1 active RPCs Aug 23 15:46:17 oss2 kernel: Lustre: 10882:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 171 previous similar messages Aug 23 15:46:17 oss2 kernel: LustreError: 10882:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810995fa9000 x1409790232205217/t0 o8->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 368/264 e 0 to 0 dl 1345758477 ref 1 fl Interpret:/0/0 rc -16/0 Aug 23 15:46:17 oss2 kernel: LustreError: 10882:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 171 previous similar messages Aug 23 15:50:14 oss2 kernel: Lustre: lustre-OST0001: slow start_page_read 10765s due to heavy IO load Aug 23 15:50:14 oss2 kernel: LustreError: 7855:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107 req@ffff8107ad101800 x1409779559983755/t0 o3->2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID:0/0 lens 448/400 e 3 to 0 dl 1345748872 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 15:50:14 oss2 kernel: Lustre: 7855:0:(ost_handler.c:887:ost_brw_read()) lustre-OST0001: ignoring bulk IO comm error with 2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID id 12345-192.168.2.227@o2ib - client will retry Aug 23 15:50:14 oss2 kernel: Lustre: 7855:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1409779559983755 took longer than estimated (1025+9742s); client may timeout. req@ffff8107ad101800 x1409779559983755/t0 o3->2b94f6e2-6ed0-1bec-352e-09fcaba969a6@NET_0x50000c0a802e3_UUID:0/0 lens 448/400 e 3 to 0 dl 1345748872 ref 1 fl Complete:/0/0 rc 0/0 Aug 23 15:50:14 oss2 kernel: Lustre: Service thread pid 7855 completed after 10766.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 15:52:52 oss2 kernel: Lustre: lustre-OST000d: slow start_page_write 157s due to heavy IO load Aug 23 15:53:37 oss2 kernel: Lustre: lustre-OST0015: slow start_page_read 11067s due to heavy IO load Aug 23 15:53:37 oss2 kernel: LustreError: 7842:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107 req@ffff810a27227400 x1409790232142161/t0 o3->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 448/400 e 5 to 0 dl 1345748785 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 15:53:37 oss2 kernel: Lustre: 7842:0:(ost_handler.c:887:ost_brw_read()) lustre-OST0015: ignoring bulk IO comm error with 78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID id 12345-192.168.2.86@o2ib - client will retry Aug 23 15:53:37 oss2 kernel: Lustre: 7842:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1409790232142161 took longer than estimated (1036+10032s); client may timeout. req@ffff810a27227400 x1409790232142161/t0 o3->78575d7a-52f5-c0fb-4a05-d9810d84195a@NET_0x50000c0a80256_UUID:0/0 lens 448/400 e 5 to 0 dl 1345748785 ref 1 fl Complete:/0/0 rc 0/0 Aug 23 15:53:37 oss2 kernel: Lustre: Service thread pid 7842 completed after 11067.64s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 16:21:53 oss2 kernel: Lustre: Failing over lustre-OST0001 Aug 23 16:21:53 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 16:21:53 oss2 kernel: LustreError: 13494:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff81052b141c00 x1392830353835590/t0 o401->@MGC192.168.64.5@o2ib_1:17/18 lens 288/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0 Aug 23 16:21:53 oss2 kernel: LustreError: 13494:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 23 previous similar messages Aug 23 16:21:53 oss2 kernel: LustreError: 7863:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff81028cabb400 x1410855903004785/t0 o3->@:0/0 lens 448/0 e 0 to 0 dl 1345760526 ref 1 fl Interpret:/0/0 rc -107/0 Aug 23 16:21:53 oss2 kernel: LustreError: 7863:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 96 previous similar messages Aug 23 16:21:53 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (stopping) Aug 23 16:21:53 oss2 kernel: LustreError: Skipped 141 previous similar messages Aug 23 16:21:57 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (stopping) Aug 23 16:21:57 oss2 kernel: LustreError: Skipped 4 previous similar messages Aug 23 16:22:05 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (stopping) Aug 23 16:22:05 oss2 kernel: LustreError: Skipped 21 previous similar messages Aug 23 16:22:11 oss2 kernel: Lustre: lustre-OST0001: shutting down for failover; client state will be preserved. Aug 23 16:22:11 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 16:22:11 oss2 kernel: Lustre: OST lustre-OST0001 has stopped. Aug 23 16:22:11 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 16:22:14 oss2 kernel: Lustre: server umount lustre-OST0001 complete Aug 23 16:22:14 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 16:22:21 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (no target) Aug 23 16:22:21 oss2 kernel: LustreError: Skipped 271 previous similar messages Aug 23 16:22:39 oss2 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Aug 23 16:22:40 oss2 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Aug 23 16:22:41 oss2 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode Aug 23 16:22:46 oss2 kernel: JBD: barrier-based sync failed on dm-6-8 - disabling barriers Aug 23 16:22:50 oss2 kernel: Lustre: 12688:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0001, 262 recoverable clients, 0 delayed clients, last_rcvd 21489571586 Aug 23 16:22:50 oss2 kernel: Lustre: lustre-OST0001: Now serving lustre-OST0001 on /dev/mapper/ost_lustre_1 with recovery enabled Aug 23 16:22:50 oss2 kernel: Lustre: lustre-OST0001: Will be in recovery for at least 5:00, or until 262 clients reconnect Aug 23 16:22:50 oss2 kernel: Lustre: 12688:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0001: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 23 16:22:59 oss2 kernel: Lustre: 25835:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0001: 261 recoverable clients remain Aug 23 16:22:59 oss2 kernel: Lustre: 25835:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 171 previous similar messages Aug 23 16:23:07 oss2 kernel: Lustre: 10850:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0001: 254 recoverable clients remain Aug 23 16:23:07 oss2 kernel: Lustre: 10850:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 6 previous similar messages Aug 23 16:24:47 oss2 kernel: Lustre: lustre-OST0001: slow parent lock 33s due to heavy IO load Aug 23 16:24:47 oss2 kernel: Lustre: lustre-OST0001: slow parent lock 33s due to heavy IO load Aug 23 16:24:50 oss2 kernel: Lustre: lustre-OST0001: slow parent lock 36s due to heavy IO load Aug 23 16:24:50 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 23 16:25:50 oss2 kernel: Lustre: 15559:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0001: 242 recoverable clients remain Aug 23 16:25:50 oss2 kernel: Lustre: 15559:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 11 previous similar messages Aug 23 16:26:06 oss2 kernel: Lustre: 13213:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0001: 241 recoverable clients remain Aug 23 16:26:39 oss2 kernel: Lustre: 5177:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0001: 207 recoverable clients remain Aug 23 16:26:39 oss2 kernel: Lustre: 5177:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 33 previous similar messages Aug 23 16:26:39 oss2 kernel: LustreError: 15516:0:(service.c:1411:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.2.123@o2ib: deadline 100+120s ago Aug 23 16:26:39 oss2 kernel: req@ffff81016c0dd000 x1409929835244681/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345760679 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 16:26:39 oss2 kernel: LustreError: 15516:0:(service.c:1411:ptlrpc_server_handle_request()) Skipped 1 previous similar message Aug 23 16:26:39 oss2 kernel: Lustre: 13176:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1409779552025908 took longer than estimated (100+120s); client may timeout. req@ffff8104c1f58c00 x1409779552025908/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345760679 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 16:26:39 oss2 kernel: Lustre: 13176:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 1 previous similar message Aug 23 16:26:39 oss2 kernel: LustreError: 5198:0:(service.c:1411:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.2.89@o2ib: deadline 100+92s ago Aug 23 16:26:39 oss2 kernel: req@ffff8105eef9a000 x1409778919207557/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1345760707 ref 1 fl Interpret:/0/0 rc 0/0 Aug 23 16:26:39 oss2 kernel: LustreError: 5198:0:(service.c:1411:ptlrpc_server_handle_request()) Skipped 300 previous similar messages Aug 23 16:26:39 oss2 kernel: Lustre: 13169:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0001: a026a4c1-f9be-e9a3-0329-be8b1c90d23a reconnecting Aug 23 16:26:39 oss2 kernel: Lustre: 13169:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 98 previous similar messages Aug 23 16:26:39 oss2 kernel: Lustre: 5200:0:(ldlm_lib.c:803:target_handle_connect()) lustre-OST0001: exp ffff8103a49f2200 already connecting Aug 23 16:26:39 oss2 kernel: LustreError: 5200:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114) req@ffff8101a9f97400 x1409778370269084/t0 o8->@:0/0 lens 368/264 e 0 to 0 dl 1345760864 ref 1 fl Interpret:/0/0 rc -114/0 Aug 23 16:26:39 oss2 kernel: LustreError: 5200:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 923 previous similar messages Aug 23 16:27:36 oss2 kernel: Lustre: lustre-OST0001: Recovery period over after 4:46, of 262 clients 261 recovered and 0 were evicted. Aug 23 16:27:36 oss2 kernel: Lustre: lustre-OST0001: sending delayed replies to recovered clients Aug 23 16:27:36 oss2 kernel: Lustre: lustre-OST0001: received MDS connection from 192.168.64.6@o2ib Aug 23 16:27:36 oss2 kernel: Lustre: 13264:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0001: deleting orphan objects from 4550585 to 4550629, orphan objids won't be reused any more. Aug 23 16:27:36 oss2 kernel: Lustre: 13264:0:(filter.c:3126:filter_destroy_precreated()) Skipped 5 previous similar messages Aug 23 17:19:04 oss2 kernel: sd 10:0:0:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Aug 23 17:19:08 oss2 kernel: sd 9:0:0:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments. Aug 23 17:59:21 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 30s due to heavy IO load Aug 23 18:13:02 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 82s due to heavy IO load Aug 23 18:13:02 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 85s due to heavy IO load Aug 23 18:16:04 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 73s due to heavy IO load Aug 23 18:29:23 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 36s due to heavy IO load Aug 23 18:30:19 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 60s due to heavy IO load Aug 23 18:30:19 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 23 18:30:46 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 118s due to heavy IO load Aug 23 18:31:10 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 50s due to heavy IO load Aug 23 18:31:58 oss2 kernel: Lustre: lustre-OST0005: slow start_page_write 48s due to heavy IO load Aug 23 18:31:58 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 151s due to heavy IO load Aug 23 19:08:58 oss2 kernel: INFO: task ll_ost_98:13267 blocked for more than 120 seconds. Aug 23 19:08:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:08:58 oss2 kernel: ll_ost_98 D ffffffff80153e4d 0 13267 1 13268 13266 (L-TLB) Aug 23 19:08:58 oss2 kernel: ffff8106112b1540 0000000000000046 ffff8106112b1550 ffffffff80062ff0 Aug 23 19:08:58 oss2 kernel: ffff810c3b26b200 000000000000000a ffff8106111cb860 ffffffff80311b60 Aug 23 19:08:58 oss2 kernel: 003de10699d9ca5c 00000000000004c3 ffff8106111cba48 000000000000288d Aug 23 19:08:58 oss2 kernel: Call Trace: Aug 23 19:08:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:08:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:08:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] __wake_up+0x38/0x4f Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:08:58 oss2 kernel: [] del_timer+0x7c/0x85 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_ast_error+0x338/0x360 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_set_transno+0x45/0x120 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_transno+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cb_interpret+0xcc/0x180 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_check_set+0x12bb/0x1580 Aug 23 19:08:58 oss2 kernel: [] try_to_del_timer_sync+0x7f/0x88 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_set_wait+0x36d/0x6d0 Aug 23 19:08:58 oss2 kernel: [] :ko2iblnd:kiblnd_send+0x86d/0x8b0 Aug 23 19:08:58 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x967/0x9b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_send_and_maybe_create_set+0x1b/0x130 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_run_bl_ast_work+0x2f5/0x3b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_process_extent_lock+0x4e2/0xad0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_add_version+0x34/0x110 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_enqueue+0x95e/0xb20 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x0/0x9b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_enqueue+0xc09/0x1210 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_handle+0x4fe3/0x55b0 Aug 23 19:08:58 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:08:58 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:08:58 oss2 kernel: Aug 23 19:08:58 oss2 kernel: INFO: task ldlm_cn_52:791 blocked for more than 120 seconds. Aug 23 19:08:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:08:58 oss2 kernel: ldlm_cn_52 D ffffffff80153e4d 0 791 1 792 790 (L-TLB) Aug 23 19:08:58 oss2 kernel: ffff810bc1fd3ae0 0000000000000046 00000000c1fd3af0 0000000000000001 Aug 23 19:08:58 oss2 kernel: 0000000300000000 0000000000000009 ffff8108997d47e0 ffff810655199100 Aug 23 19:08:58 oss2 kernel: 003de10695e18de9 0000000000004a9a ffff8108997d49c8 0000000300000082 Aug 23 19:08:58 oss2 kernel: Call Trace: Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:08:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:08:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:08:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_export_lock_put+0x6f/0xe0 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:08:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:08:58 oss2 kernel: Aug 23 19:08:58 oss2 kernel: INFO: task ldlm_cn_64:803 blocked for more than 120 seconds. Aug 23 19:08:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:08:58 oss2 kernel: ldlm_cn_64 D ffffffff80153e4d 0 803 1 804 802 (L-TLB) Aug 23 19:08:58 oss2 kernel: ffff81077d2cfae0 0000000000000046 000000007d2cfaf0 0000000000000001 Aug 23 19:08:58 oss2 kernel: 0000000500000000 0000000000000009 ffff810acf611860 ffff810655232080 Aug 23 19:08:58 oss2 kernel: 003de10656f36e6b 00000000000017fb ffff810acf611a48 0000000500000082 Aug 23 19:08:58 oss2 kernel: Call Trace: Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:08:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:08:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:08:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_export_lock_put+0x6f/0xe0 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:08:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:08:58 oss2 kernel: Aug 23 19:08:58 oss2 kernel: INFO: task ldlm_cn_72:811 blocked for more than 120 seconds. Aug 23 19:08:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:08:58 oss2 kernel: ldlm_cn_72 D ffffffff80153e4d 0 811 1 812 810 (L-TLB) Aug 23 19:08:58 oss2 kernel: ffff81044d403ae0 0000000000000046 ffff81044d403af0 ffffffff80062ff0 Aug 23 19:08:58 oss2 kernel: 0000001100000000 000000000000000a ffff81053dbb4860 ffff810c3fd01080 Aug 23 19:08:58 oss2 kernel: 003de10699dc5a48 00000000000007fe ffff81053dbb4a48 00000011000005f3 Aug 23 19:08:58 oss2 kernel: Call Trace: Aug 23 19:08:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:08:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:08:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:08:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:08:58 oss2 kernel: [] :lnet:LNetMDBind+0x301/0x450 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:08:58 oss2 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Aug 23 19:08:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:08:58 oss2 kernel: Aug 23 19:08:58 oss2 kernel: INFO: task ll_ost_128:10804 blocked for more than 120 seconds. Aug 23 19:08:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:08:58 oss2 kernel: ll_ost_128 D ffffffff80153e4d 0 10804 1 10805 3212 (L-TLB) Aug 23 19:08:58 oss2 kernel: ffff8101ea199970 0000000000000046 ffff8101ea199980 ffffffff80062ff0 Aug 23 19:08:58 oss2 kernel: 0000000000000282 000000000000000a ffff8101929c3100 ffff810655342040 Aug 23 19:08:58 oss2 kernel: 003de109510ee6ee 0000000000000638 ffff8101929c32e8 0000000800008840 Aug 23 19:08:58 oss2 kernel: Call Trace: Aug 23 19:08:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:08:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:08:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:08:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_server_glimpse_ast+0x30e/0x3b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:interval_iterate_reverse+0x73/0x240 Aug 23 19:08:58 oss2 kernel: [] :lnet:lnet_prep_send+0x67/0xb0 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:08:58 oss2 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Aug 23 19:08:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_server_glimpse_ast+0x0/0x3b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_server_completion_ast+0x0/0x5e0 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_blocking_ast+0x0/0x9b0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ldlm_handle_enqueue+0x1d6/0x1210 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Aug 23 19:08:58 oss2 kernel: [] :ost:ost_handle+0x4fe3/0x55b0 Aug 23 19:08:58 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:08:58 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:08:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:08:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:08:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:08:58 oss2 kernel: Aug 23 19:09:39 oss2 kernel: Lustre: Service thread pid 13267 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:09:39 oss2 kernel: Pid: 13267, comm: ll_ost_98 Aug 23 19:09:39 oss2 kernel: Aug 23 19:09:39 oss2 kernel: Call Trace: Aug 23 19:09:39 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:09:39 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:09:39 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:09:39 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:09:39 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:09:39 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:09:39 oss2 kernel: [] __wake_up+0x38/0x4f Aug 23 19:09:39 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:09:39 oss2 kernel: [] del_timer+0x7c/0x85 Aug 23 19:09:39 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_handle_ast_error+0x338/0x360 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] lustre_msg_set_transno+0x45/0x120 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] lustre_msg_get_transno+0x35/0xf0 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_cb_interpret+0xcc/0x180 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ptlrpc_check_set+0x12bb/0x1580 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] try_to_del_timer_sync+0x7f/0x88 Aug 23 19:09:39 oss2 kernel: [] ptlrpc_set_wait+0x36d/0x6d0 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Aug 23 19:09:39 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 19:09:39 oss2 kernel: [] ost_blocking_ast+0x967/0x9b0 [ost] Aug 23 19:09:39 oss2 kernel: [] ldlm_send_and_maybe_create_set+0x1b/0x130 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_run_bl_ast_work+0x2f5/0x3b0 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_process_extent_lock+0x4e2/0xad0 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] lustre_msg_add_version+0x34/0x110 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ldlm_lock_enqueue+0x95e/0xb20 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Aug 23 19:09:39 oss2 kernel: [] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Aug 23 19:09:39 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:09:39 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:09:39 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:09:39 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:09:39 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:09:39 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:09:39 oss2 kernel: Aug 23 19:09:39 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770579.13267 Aug 23 19:09:50 oss2 kernel: Lustre: Service thread pid 10804 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:09:50 oss2 kernel: Pid: 10804, comm: ll_ost_128 Aug 23 19:09:50 oss2 kernel: Aug 23 19:09:50 oss2 kernel: Call Trace: Aug 23 19:09:50 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:09:50 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:09:50 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:09:50 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:09:50 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:09:50 oss2 kernel: [] ldlm_server_glimpse_ast+0x30e/0x3b0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] interval_iterate_reverse+0x73/0x240 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Aug 23 19:09:50 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:09:50 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 19:09:50 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:09:50 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ldlm_request_cancel+0x265/0x330 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] lustre_swab_buf+0x81/0x170 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Aug 23 19:09:50 oss2 kernel: [] ldlm_handle_enqueue+0x1d6/0x1210 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Aug 23 19:09:50 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:09:50 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:09:50 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:09:50 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:09:50 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:09:50 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:09:50 oss2 kernel: Aug 23 19:09:50 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770590.10804 Aug 23 19:10:28 oss2 kernel: Lustre: Service thread pid 13265 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:10:28 oss2 kernel: Pid: 13265, comm: ll_ost_96 Aug 23 19:10:28 oss2 kernel: Aug 23 19:10:28 oss2 kernel: Call Trace: Aug 23 19:10:28 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:10:28 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:10:28 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:28 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:10:28 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:10:28 oss2 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Aug 23 19:10:28 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:10:28 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 19:10:28 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:28 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ldlm_request_cancel+0x265/0x330 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] lustre_swab_buf+0x81/0x170 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Aug 23 19:10:28 oss2 kernel: [] ldlm_handle_enqueue+0x1d6/0x1210 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Aug 23 19:10:28 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:10:28 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:10:28 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:28 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:28 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:10:28 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:28 oss2 kernel: Aug 23 19:10:28 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770628.13265 Aug 23 19:10:38 oss2 kernel: Lustre: Service thread pid 18459 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:10:38 oss2 kernel: Pid: 18459, comm: ll_ost_257 Aug 23 19:10:38 oss2 kernel: Aug 23 19:10:38 oss2 kernel: Call Trace: Aug 23 19:10:38 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:10:38 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:10:38 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:38 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:10:38 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:10:38 oss2 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Aug 23 19:10:38 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:10:38 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 19:10:38 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:38 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ldlm_request_cancel+0x265/0x330 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] lustre_swab_buf+0x81/0x170 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Aug 23 19:10:38 oss2 kernel: [] ldlm_handle_enqueue+0x1d6/0x1210 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Aug 23 19:10:38 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:10:38 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:10:38 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:38 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:38 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:10:38 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:38 oss2 kernel: Aug 23 19:10:38 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770638.18459 Aug 23 19:10:58 oss2 kernel: INFO: task ll_ost_96:13265 blocked for more than 120 seconds. Aug 23 19:10:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:10:58 oss2 kernel: ll_ost_96 D ffffffff80153e4d 0 13265 1 13266 13264 (L-TLB) Aug 23 19:10:58 oss2 kernel: ffff81061129f970 0000000000000046 ffff81061129f980 ffffffff80062ff0 Aug 23 19:10:58 oss2 kernel: ffff8103fba59170 000000000000000a ffff8106111c9080 ffff810c3fcef040 Aug 23 19:10:58 oss2 kernel: 003de112178272d4 0000000000000776 ffff8106111c9268 00000010000055f6 Aug 23 19:10:58 oss2 kernel: Call Trace: Aug 23 19:10:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:10:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:10:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:10:58 oss2 kernel: [] :lnet:lnet_prep_send+0x67/0xb0 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:10:58 oss2 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Aug 23 19:10:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_swab_buf+0x81/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_server_glimpse_ast+0x0/0x3b0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_server_completion_ast+0x0/0x5e0 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x0/0x9b0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_enqueue+0x1d6/0x1210 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_version+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_handle+0x4fe3/0x55b0 Aug 23 19:10:58 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:10:58 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:58 oss2 kernel: Aug 23 19:10:58 oss2 kernel: INFO: task ll_ost_98:13267 blocked for more than 120 seconds. Aug 23 19:10:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:10:58 oss2 kernel: ll_ost_98 D ffffffff80153e4d 0 13267 1 13268 13266 (L-TLB) Aug 23 19:10:58 oss2 kernel: ffff8106112b1540 0000000000000046 ffff8106112b1550 ffffffff80062ff0 Aug 23 19:10:58 oss2 kernel: ffff810c3b26b200 000000000000000a ffff8106111cb860 ffffffff80311b60 Aug 23 19:10:58 oss2 kernel: 003de10699d9ca5c 00000000000004c3 ffff8106111cba48 000000000000288d Aug 23 19:10:58 oss2 kernel: Call Trace: Aug 23 19:10:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:10:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:10:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] __wake_up+0x38/0x4f Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:10:58 oss2 kernel: [] del_timer+0x7c/0x85 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_ast_error+0x338/0x360 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_set_transno+0x45/0x120 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_transno+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cb_interpret+0xcc/0x180 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_check_set+0x12bb/0x1580 Aug 23 19:10:58 oss2 kernel: [] try_to_del_timer_sync+0x7f/0x88 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_set_wait+0x36d/0x6d0 Aug 23 19:10:58 oss2 kernel: [] :ko2iblnd:kiblnd_send+0x86d/0x8b0 Aug 23 19:10:58 oss2 kernel: [] default_wake_function+0x0/0xe Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x967/0x9b0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_send_and_maybe_create_set+0x1b/0x130 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_run_bl_ast_work+0x2f5/0x3b0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_process_extent_lock+0x4e2/0xad0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_add_version+0x34/0x110 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_enqueue+0x95e/0xb20 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x0/0x9b0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_enqueue+0xc09/0x1210 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_handle+0x4fe3/0x55b0 Aug 23 19:10:58 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:10:58 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:58 oss2 kernel: Aug 23 19:10:58 oss2 kernel: INFO: task ldlm_cn_52:791 blocked for more than 120 seconds. Aug 23 19:10:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:10:58 oss2 kernel: ldlm_cn_52 D ffffffff80153e4d 0 791 1 792 790 (L-TLB) Aug 23 19:10:58 oss2 kernel: ffff810bc1fd3ae0 0000000000000046 00000000c1fd3af0 0000000000000001 Aug 23 19:10:58 oss2 kernel: 0000000300000000 0000000000000009 ffff8108997d47e0 ffff810655199100 Aug 23 19:10:58 oss2 kernel: 003de10695e18de9 0000000000004a9a ffff8108997d49c8 0000000300000082 Aug 23 19:10:58 oss2 kernel: Call Trace: Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:10:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:10:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:10:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_export_lock_put+0x6f/0xe0 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:10:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:58 oss2 kernel: Aug 23 19:10:58 oss2 kernel: INFO: task ldlm_cn_64:803 blocked for more than 120 seconds. Aug 23 19:10:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:10:58 oss2 kernel: ldlm_cn_64 D ffffffff80153e4d 0 803 1 804 802 (L-TLB) Aug 23 19:10:58 oss2 kernel: ffff81077d2cfae0 0000000000000046 000000007d2cfaf0 0000000000000001 Aug 23 19:10:58 oss2 kernel: 0000000500000000 0000000000000009 ffff810acf611860 ffff810655232080 Aug 23 19:10:58 oss2 kernel: 003de10656f36e6b 00000000000017fb ffff810acf611a48 0000000500000082 Aug 23 19:10:58 oss2 kernel: Call Trace: Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:10:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:10:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:10:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_export_lock_put+0x6f/0xe0 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:10:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:58 oss2 kernel: Aug 23 19:10:58 oss2 kernel: INFO: task ldlm_cn_72:811 blocked for more than 120 seconds. Aug 23 19:10:58 oss2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 23 19:10:58 oss2 kernel: ldlm_cn_72 D ffffffff80153e4d 0 811 1 812 810 (L-TLB) Aug 23 19:10:58 oss2 kernel: ffff81044d403ae0 0000000000000046 ffff81044d403af0 ffffffff80062ff0 Aug 23 19:10:58 oss2 kernel: 0000001100000000 000000000000000a ffff81053dbb4860 ffff810c3fd01080 Aug 23 19:10:58 oss2 kernel: 003de10699dc5a48 00000000000007fe ffff81053dbb4a48 00000011000005f3 Aug 23 19:10:58 oss2 kernel: Call Trace: Aug 23 19:10:58 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_log_wait_commit+0xa3/0xf5 Aug 23 19:10:58 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:10:58 oss2 kernel: [] :jbd2:jbd2_journal_stop+0x1e7/0x216 Aug 23 19:10:58 oss2 kernel: [] :obdfilter:filter_sync+0xc5/0x5c0 Aug 23 19:10:58 oss2 kernel: [] :lnet:LNetPut+0x730/0x840 Aug 23 19:10:58 oss2 kernel: [] :lnet:LNetMDBind+0x301/0x450 Aug 23 19:10:58 oss2 kernel: [] :ost:ost_blocking_ast+0x79c/0x9b0 Aug 23 19:10:58 oss2 kernel: [] :obdclass:class_handle2object+0xe0/0x170 Aug 23 19:10:58 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_callback+0x6d/0xd0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_lock_cancel+0xc0/0x170 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_request_cancel+0x265/0x330 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_pack_reply+0x29/0xb0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ldlm_cancel_handler+0x308/0x690 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:lustre_msg_get_opc+0x35/0xf0 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310 Aug 23 19:10:58 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0xf66/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:10:58 oss2 kernel: [] :ptlrpc:ptlrpc_main+0x0/0x1120 Aug 23 19:10:58 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:10:58 oss2 kernel: Aug 23 19:14:14 oss2 kernel: Lustre: Service thread pid 13263 was inactive for 262.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:14:14 oss2 kernel: Pid: 13263, comm: ll_ost_94 Aug 23 19:14:14 oss2 kernel: Aug 23 19:14:14 oss2 kernel: Call Trace: Aug 23 19:14:14 oss2 kernel: [] thread_return+0x62/0xfe Aug 23 19:14:14 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:14:14 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:14:14 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:14:14 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:14:14 oss2 kernel: [] ldlm_pool_add+0x131/0x190 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] cache_alloc_refill+0x86/0x186 Aug 23 19:14:14 oss2 kernel: [] lnet_prep_send+0x67/0xb0 [lnet] Aug 23 19:14:14 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:14:14 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 23 19:14:14 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:14:14 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ldlm_request_cancel+0x265/0x330 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] lustre_swab_buf+0x81/0x170 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ldlm_server_completion_ast+0x0/0x5e0 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Aug 23 19:14:14 oss2 kernel: [] ldlm_handle_enqueue+0x1d6/0x1210 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ost_handle+0x4fe3/0x55b0 [ost] Aug 23 19:14:14 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 23 19:14:14 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 23 19:14:14 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:14:14 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:14:14 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:14:14 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:14:14 oss2 kernel: Aug 23 19:14:14 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770854.13263 Aug 23 19:16:14 oss2 kernel: Lustre: Service thread pid 13267 completed after 595.54s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 19:16:14 oss2 kernel: Lustre: Service thread pid 13263 completed after 382.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 23 19:16:14 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 23 19:16:14 oss2 kernel: Lustre: Service thread pid 803 was inactive for 600.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 23 19:16:14 oss2 kernel: Pid: 803, comm: ldlm_cn_64 Aug 23 19:16:14 oss2 kernel: Aug 23 19:16:14 oss2 kernel: Call Trace: Aug 23 19:16:14 oss2 kernel: [] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Aug 23 19:16:14 oss2 kernel: [] autoremove_wake_function+0x0/0x2e Aug 23 19:16:14 oss2 kernel: [] jbd2_journal_stop+0x1e7/0x216 [jbd2] Aug 23 19:16:14 oss2 kernel: [] filter_sync+0xc5/0x5c0 [obdfilter] Aug 23 19:16:14 oss2 kernel: [] LNetPut+0x730/0x840 [lnet] Aug 23 19:16:14 oss2 kernel: [] ldlm_export_lock_put+0x6f/0xe0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ost_blocking_ast+0x79c/0x9b0 [ost] Aug 23 19:16:14 oss2 kernel: [] _spin_lock_bh+0x9/0x14 Aug 23 19:16:14 oss2 kernel: [] ldlm_cancel_callback+0x6d/0xd0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ldlm_lock_cancel+0xc0/0x170 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ldlm_request_cancel+0x265/0x330 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] lustre_pack_reply+0x29/0xb0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ldlm_handle_cancel+0x21d/0x2e0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ldlm_cancel_handler+0x308/0x690 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 23 19:16:14 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] child_rip+0xa/0x11 Aug 23 19:16:14 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 23 19:16:14 oss2 kernel: [] child_rip+0x0/0x11 Aug 23 19:16:14 oss2 kernel: Aug 23 19:16:14 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1345770974.803 Aug 23 21:49:33 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 71s due to heavy IO load Aug 23 21:49:38 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 71s due to heavy IO load Aug 23 22:43:11 oss2 kernel: Lustre: lustre-OST000d: slow start_page_write 51s due to heavy IO load Aug 23 22:43:12 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 53s due to heavy IO load Aug 23 23:04:43 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 60s due to heavy IO load Aug 24 09:40:03 oss2 kernel: Lustre: lustre-OST0005: slow start_page_write 66s due to heavy IO load Aug 24 09:40:03 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 69s due to heavy IO load Aug 24 09:58:30 oss2 kernel: Lustre: lustre-OST0009: slow start_page_read 57s due to heavy IO load Aug 24 09:58:44 oss2 kernel: Lustre: lustre-OST0009: slow start_page_read 71s due to heavy IO load Aug 24 09:59:26 oss2 kernel: Lustre: lustre-OST0009: slow start_page_read 41s due to heavy IO load Aug 24 09:59:26 oss2 kernel: Lustre: lustre-OST0009: slow start_page_read 41s due to heavy IO load Aug 24 10:09:00 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 53s due to heavy IO load Aug 24 10:35:42 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 31s due to heavy IO load Aug 25 21:31:12 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 43s due to heavy IO load Aug 25 21:31:12 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 25 21:50:52 oss2 kernel: Lustre: Failing over lustre-OST0015 Aug 25 21:50:53 oss2 kernel: LustreError: 5180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810be085f400 x1410855904046628/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1345953069 ref 1 fl Interpret:H/0/0 rc -107/0 Aug 25 21:50:53 oss2 kernel: LustreError: 5180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Aug 25 21:50:53 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0015_UUID' is not available for connect (stopping) Aug 25 21:50:53 oss2 kernel: LustreError: Skipped 471 previous similar messages Aug 25 21:50:57 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0005_UUID' is not available for connect (stopping) Aug 25 21:50:57 oss2 kernel: LustreError: Skipped 45 previous similar messages Aug 25 21:51:05 oss2 kernel: Lustre: lustre-OST0005: shutting down for failover; client state will be preserved. Aug 25 21:51:05 oss2 kernel: Lustre: OST lustre-OST0005 has stopped. Aug 25 21:51:05 oss2 kernel: Lustre: server umount lustre-OST0005 complete Aug 25 21:51:05 oss2 kernel: Lustre: lustre-OST0015: shutting down for failover; client state will be preserved. Aug 25 21:51:05 oss2 kernel: Lustre: OST lustre-OST0015 has stopped. Aug 25 21:51:05 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0005_UUID' is not available for connect (no target) Aug 25 21:51:05 oss2 kernel: LustreError: Skipped 119 previous similar messages Aug 25 21:51:06 oss2 kernel: Lustre: server umount lustre-OST0015 complete Aug 25 21:51:07 oss2 kernel: LDISKFS-fs (dm-26): warning: maximal mount count reached, running e2fsck is recommended Aug 25 21:51:07 oss2 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Aug 25 21:51:07 oss2 kernel: LDISKFS-fs (dm-26): mounted filesystem with ordered data mode Aug 25 21:51:07 oss2 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Aug 25 21:51:07 oss2 kernel: JBD: barrier-based sync failed on dm-26-8 - disabling barriers Aug 25 21:51:07 oss2 kernel: LDISKFS-fs (dm-26): warning: maximal mount count reached, running e2fsck is recommended Aug 25 21:51:08 oss2 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode Aug 25 21:51:08 oss2 kernel: LDISKFS-fs (dm-26): mounted filesystem with ordered data mode Aug 25 21:51:08 oss2 kernel: Lustre: 18813:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0005, 262 recoverable clients, 0 delayed clients, last_rcvd 21490464977 Aug 25 21:51:08 oss2 kernel: JBD: barrier-based sync failed on dm-10-8 - disabling barriers Aug 25 21:51:08 oss2 kernel: Lustre: lustre-OST0005: Now serving lustre-OST0005 on /dev/mapper/ost_lustre_5 with recovery enabled Aug 25 21:51:08 oss2 kernel: Lustre: lustre-OST0005: Will be in recovery for at least 5:00, or until 262 clients reconnect Aug 25 21:51:08 oss2 kernel: Lustre: 18813:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0005: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 25 21:51:08 oss2 kernel: Lustre: 18827:0:(filter.c:1001:filter_init_server_data()) RECOVERY: service lustre-OST0015, 262 recoverable clients, 0 delayed clients, last_rcvd 25786265288 Aug 25 21:51:08 oss2 kernel: JBD: barrier-based sync failed on dm-26-8 - disabling barriers Aug 25 21:51:08 oss2 kernel: Lustre: lustre-OST0015: Now serving lustre-OST0015 on /dev/mapper/ost_lustre_21 with recovery enabled Aug 25 21:51:08 oss2 kernel: Lustre: lustre-OST0015: Will be in recovery for at least 5:00, or until 262 clients reconnect Aug 25 21:51:08 oss2 kernel: Lustre: 18827:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lustre-OST0015: quotaon failed because quota files don't exist, please run quotacheck firstly Aug 25 21:51:08 oss2 kernel: Lustre: 10868:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0005: 261 recoverable clients remain Aug 25 21:51:08 oss2 kernel: Lustre: 10868:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 205 previous similar messages Aug 25 21:51:12 oss2 kernel: LustreError: 13225:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107) req@ffff810b2c502000 x1409778236485865/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1345953088 ref 1 fl Interpret:H/0/0 rc -107/0 Aug 25 21:51:12 oss2 kernel: LustreError: 13225:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 544 previous similar messages Aug 25 21:51:16 oss2 kernel: Lustre: 15518:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) lustre-OST0015: 82 recoverable clients remain Aug 25 21:51:16 oss2 kernel: Lustre: 15518:0:(ldlm_lib.c:1815:target_queue_last_replay_reply()) Skipped 356 previous similar messages Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0015: Recovery period over after 0:16, of 262 clients 261 recovered and 0 were evicted. Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0015: sending delayed replies to recovered clients Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0015: received MDS connection from 192.168.64.6@o2ib Aug 25 21:51:24 oss2 kernel: Lustre: 16660:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0015: deleting orphan objects from 4763177 to 4766017, orphan objids won't be reused any more. Aug 25 21:51:24 oss2 kernel: LustreError: 4529:0:(filter.c:1557:filter_destroy_internal()) destroying objid 4763150 ino 55267561 nlink 0 count 2 Aug 25 21:51:24 oss2 kernel: LustreError: 4529:0:(filter.c:1563:filter_destroy_internal()) error unlinking objid 4763150: rc -2 Aug 25 21:51:24 oss2 kernel: LustreError: 18952:0:(filter.c:1557:filter_destroy_internal()) destroying objid 4763165 ino 55267584 nlink 0 count 2 Aug 25 21:51:24 oss2 kernel: LustreError: 18952:0:(filter.c:1563:filter_destroy_internal()) error unlinking objid 4763165: rc -2 Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0005: Recovery period over after 0:16, of 262 clients 262 recovered and 0 were evicted. Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0005: sending delayed replies to recovered clients Aug 25 21:51:24 oss2 kernel: Lustre: lustre-OST0005: received MDS connection from 192.168.64.6@o2ib Aug 25 21:51:24 oss2 kernel: Lustre: 13249:0:(filter.c:3126:filter_destroy_precreated()) lustre-OST0005: deleting orphan objects from 4763207 to 4764801, orphan objids won't be reused any more. Aug 25 21:51:24 oss2 kernel: LustreError: 18957:0:(filter.c:1557:filter_destroy_internal()) destroying objid 4763143 ino 31503176 nlink 0 count 2 Aug 25 21:51:24 oss2 kernel: LustreError: 18957:0:(filter.c:1557:filter_destroy_internal()) Skipped 1 previous similar message Aug 25 21:51:24 oss2 kernel: LustreError: 18957:0:(filter.c:1563:filter_destroy_internal()) error unlinking objid 4763143: rc -2 Aug 25 21:51:24 oss2 kernel: LustreError: 18957:0:(filter.c:1563:filter_destroy_internal()) Skipped 1 previous similar message Aug 25 22:41:36 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 39s due to heavy IO load Aug 25 22:43:23 oss2 kernel: Lustre: lustre-OST0005: slow start_page_write 43s due to heavy IO load Aug 25 22:43:43 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 75s due to heavy IO load Aug 25 22:43:43 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 26 00:07:16 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 41s due to heavy IO load Aug 26 00:07:16 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 41s due to heavy IO load Aug 26 01:47:19 oss2 kernel: Lustre: lustre-OST0005: slow start_page_write 49s due to heavy IO load Aug 26 01:48:17 oss2 kernel: Lustre: lustre-OST0009: slow preprw_read setup 44s due to heavy IO load Aug 26 01:48:17 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 44s due to heavy IO load Aug 26 01:48:17 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 26 03:22:38 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 117s due to heavy IO load Aug 26 03:23:23 oss2 kernel: Lustre: lustre-OST0011: slow start_page_write 44s due to heavy IO load Aug 26 03:25:47 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 126s due to heavy IO load Aug 26 03:27:34 oss2 kernel: Lustre: lustre-OST0001: slow start_page_write 45s due to heavy IO load Aug 26 03:33:35 oss2 kernel: Lustre: lustre-OST0005: slow start_page_write 47s due to heavy IO load Aug 26 03:52:38 oss2 kernel: Lustre: lustre-OST0015: slow start_page_write 198s due to heavy IO load Aug 26 03:52:38 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 26 03:56:44 oss2 kernel: Lustre: lustre-OST0009: slow start_page_write 76s due to heavy IO load Aug 26 03:56:44 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 26 05:01:08 oss2 kernel: Lustre: lustre-OST000d: slow start_page_write 31s due to heavy IO load Aug 27 03:22:11 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 03:22:11 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 27 03:57:10 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0018_UUID' is not available for connect (no target) Aug 27 03:57:10 oss2 kernel: LustreError: Skipped 109 previous similar messages Aug 27 03:57:10 oss2 kernel: LustreError: 18412:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8108cc214400 x1409790705044844/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346061530 ref 1 fl Interpret:/0/0 rc -19/0 Aug 27 03:57:10 oss2 kernel: LustreError: 18412:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 72 previous similar messages Aug 27 04:09:16 oss2 kernel: Lustre: lustre-OST0005: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 04:09:16 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 27 04:55:56 oss2 kernel: Lustre: lustre-OST000d: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 04:55:56 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 27 04:55:56 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 04:55:56 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 27 05:03:25 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0008_UUID' is not available for connect (no target) Aug 27 05:03:25 oss2 kernel: LustreError: 18484:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106143f4850 x1409790705054283/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346065505 ref 1 fl Interpret:/0/0 rc -19/0 Aug 27 05:33:01 oss2 kernel: Lustre: lustre-OST0009: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 05:33:01 oss2 kernel: Lustre: Skipped 2 previous similar messages Aug 27 05:34:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Aug 27 05:34:03 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 27 05:34:03 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8104f7a42c00 x1409790705058589/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346067343 ref 1 fl Interpret:/0/0 rc -19/0 Aug 27 05:34:03 oss2 kernel: LustreError: 13219:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 27 05:58:26 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client c29dee99-04ac-437e-39e3-57d93e27263b (at 192.168.2.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 05:58:26 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 27 06:05:56 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 27 06:05:56 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 27 06:05:56 oss2 kernel: LustreError: 5237:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8107ca9dc400 x1409790705062881/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346069256 ref 1 fl Interpret:/0/0 rc -19/0 Aug 27 06:05:56 oss2 kernel: LustreError: 5237:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 27 15:15:17 oss2 kernel: Lustre: lustre-OST0001: haven't heard from client a041b23e-a19b-ed75-454b-c5411c077fdc (at 192.168.11.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 15:15:17 oss2 kernel: Lustre: Skipped 5 previous similar messages Aug 27 15:15:17 oss2 kernel: Lustre: lustre-OST0011: haven't heard from client a041b23e-a19b-ed75-454b-c5411c077fdc (at 192.168.11.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Aug 27 15:15:17 oss2 kernel: Lustre: Skipped 1 previous similar message Aug 28 04:39:39 oss2 kernel: LustreError: 10820:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0015_UUID: lvbo_init failed for resource 4806412: rc -2 Aug 28 04:39:39 oss2 kernel: LustreError: 15504:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0011_UUID: lvbo_init failed for resource 4803369: rc -2 Aug 28 04:41:42 oss2 kernel: LustreError: 16664:0:(ldlm_resource.c:862:ldlm_resource_add()) filter-lustre-OST0001_UUID: lvbo_init failed for resource 4803195: rc -2 Aug 28 08:36:59 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:36:59 oss2 kernel: LustreError: Skipped 6 previous similar messages Aug 28 08:36:59 oss2 kernel: LustreError: 25842:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810691659400 x1409779468012566/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164719 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:36:59 oss2 kernel: LustreError: 25842:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages Aug 28 08:36:59 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:36:59 oss2 kernel: LustreError: 5224:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810691659800 x1409778369641986/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164719 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:00 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:37:00 oss2 kernel: LustreError: Skipped 22 previous similar messages Aug 28 08:37:00 oss2 kernel: LustreError: 10872:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8101d0be1000 x1409779397299914/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164720 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:00 oss2 kernel: LustreError: 10872:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 22 previous similar messages Aug 28 08:37:01 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:37:01 oss2 kernel: LustreError: Skipped 48 previous similar messages Aug 28 08:37:01 oss2 kernel: LustreError: 5212:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff81074e54a800 x1409779257148130/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164721 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:01 oss2 kernel: LustreError: 5212:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 48 previous similar messages Aug 28 08:37:03 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:37:03 oss2 kernel: LustreError: Skipped 57 previous similar messages Aug 28 08:37:03 oss2 kernel: LustreError: 10882:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103dfa30c00 x1409066086007191/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164723 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:03 oss2 kernel: LustreError: 10882:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 57 previous similar messages Aug 28 08:37:07 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:37:07 oss2 kernel: LustreError: Skipped 58 previous similar messages Aug 28 08:37:07 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff8109b5ec8000 x1409779464949176/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164727 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:07 oss2 kernel: LustreError: 13266:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 58 previous similar messages Aug 28 08:37:15 oss2 kernel: LustreError: 137-5: UUID 'lustre-OST000c_UUID' is not available for connect (no target) Aug 28 08:37:15 oss2 kernel: LustreError: Skipped 68 previous similar messages Aug 28 08:37:15 oss2 kernel: LustreError: 10856:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19) req@ffff810207813800 x1409789564450891/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1346164735 ref 1 fl Interpret:/0/0 rc -19/0 Aug 28 08:37:15 oss2 kernel: LustreError: 10856:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 68 previous similar messages Aug 28 09:31:37 oss2 kernel: Lustre: lustre-OST0001: slow start_page_read 30s due to heavy IO load Aug 28 09:41:46 oss2 kernel: Lustre: Service thread pid 13381 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 28 09:41:46 oss2 kernel: Pid: 13381, comm: ll_ost_io_82 Aug 28 09:41:46 oss2 kernel: Aug 28 09:41:46 oss2 kernel: Call Trace: Aug 28 09:41:46 oss2 kernel: [] lcw_cb+0x0/0x460 [libcfs] Aug 28 09:41:46 oss2 kernel: [] libcfs_debug_dumpstack+0x51/0x60 [libcfs] Aug 28 09:41:46 oss2 kernel: [] lcw_cb+0x33c/0x460 [libcfs] Aug 28 09:41:46 oss2 kernel: [] run_timer_softirq+0x193/0x241 Aug 28 09:41:46 oss2 kernel: [] __do_softirq+0x89/0x133 Aug 28 09:41:46 oss2 kernel: [] call_softirq+0x1c/0x28 Aug 28 09:41:46 oss2 kernel: [] do_softirq+0x2c/0x7d Aug 28 09:41:46 oss2 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 28 09:41:46 oss2 kernel: [] _spin_unlock_irqrestore+0x8/0x9 Aug 28 09:41:46 oss2 kernel: [] __down_trylock+0x44/0x4e Aug 28 09:41:46 oss2 kernel: [] __down_failed_trylock+0x35/0x3a Aug 28 09:41:46 oss2 kernel: [] ldiskfs_destroy_inode+0x0/0x90 [ldiskfs] Aug 28 09:41:46 oss2 kernel: [] .text.lock.ldlm_resource+0x73/0x87 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] ldlm_pools_shrink+0x147/0x2f0 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] __down_read+0x12/0x92 Aug 28 09:41:46 oss2 kernel: [] __up_read+0x19/0x7f Aug 28 09:41:46 oss2 kernel: [] shrink_slab+0xdc/0x153 Aug 28 09:41:46 oss2 kernel: [] zone_reclaim+0x235/0x2cd Aug 28 09:41:46 oss2 kernel: [] __rmqueue+0x44/0xc6 Aug 28 09:41:46 oss2 kernel: [] get_page_from_freelist+0xbf/0x43a Aug 28 09:41:46 oss2 kernel: [] __alloc_pages+0x78/0x308 Aug 28 09:41:46 oss2 kernel: [] find_or_create_page+0x32/0x72 Aug 28 09:41:46 oss2 kernel: [] filter_get_page+0x35/0x70 [obdfilter] Aug 28 09:41:46 oss2 kernel: [] filter_preprw+0x14da/0x1e00 [obdfilter] Aug 28 09:41:46 oss2 kernel: [] lnet_ni_send+0x96/0xe0 [lnet] Aug 28 09:41:46 oss2 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Aug 28 09:41:46 oss2 kernel: [] lnet_send+0x9a3/0x9d0 [lnet] Aug 28 09:41:46 oss2 kernel: [] ost_brw_write+0xf9c/0x2480 [ost] Aug 28 09:41:46 oss2 kernel: [] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] target_committed_to_req+0x40/0x120 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] default_wake_function+0x0/0xe Aug 28 09:41:46 oss2 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] ost_handle+0x2bae/0x55b0 [ost] Aug 28 09:41:46 oss2 kernel: [] __next_cpu+0x19/0x28 Aug 28 09:41:46 oss2 kernel: [] smp_send_reschedule+0x4e/0x53 Aug 28 09:41:46 oss2 kernel: [] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] __wake_up_common+0x3e/0x68 Aug 28 09:41:46 oss2 kernel: [] ptlrpc_main+0xf66/0x1120 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] child_rip+0xa/0x11 Aug 28 09:41:46 oss2 kernel: [] ptlrpc_main+0x0/0x1120 [ptlrpc] Aug 28 09:41:46 oss2 kernel: [] child_rip+0x0/0x11 Aug 28 09:41:46 oss2 kernel: Aug 28 09:41:46 oss2 kernel: LustreError: dumping log to /tmp/lustre-log.1346168506.13381 Aug 28 09:54:33 oss2 kernel: Lustre: 13371:0:(service.c:808:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply Aug 28 09:54:33 oss2 kernel: req@ffff8102aaf8a800 x1409779385201543/t0 o4->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 448/416 e 4 to 0 dl 1346169278 ref 2 fl Interpret:/0/0 rc 0/0 Aug 28 09:56:49 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 09:56:49 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 573 previous similar messages Aug 28 09:56:49 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 09:56:49 oss2 kernel: Lustre: 13296:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 96 previous similar messages Aug 28 09:56:49 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810a844e2000 x1409779385554423/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169509 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:56:49 oss2 kernel: LustreError: 13296:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 212 previous similar messages Aug 28 09:56:56 oss2 kernel: LustreError: 13254:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3de85400 x1409779385555734/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169516 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:57:03 oss2 kernel: Lustre: 5203:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 09:57:03 oss2 kernel: Lustre: 5203:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message Aug 28 09:57:03 oss2 kernel: LustreError: 5203:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8107b70c9000 x1409779385557059/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169523 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:57:17 oss2 kernel: LustreError: 10814:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106088b8000 x1409779385559864/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169537 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:57:17 oss2 kernel: LustreError: 10814:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message Aug 28 09:57:24 oss2 kernel: Lustre: 25860:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 09:57:24 oss2 kernel: Lustre: 25860:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages Aug 28 09:57:38 oss2 kernel: LustreError: 13291:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8109a4146800 x1409779385563715/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169558 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:57:38 oss2 kernel: LustreError: 13291:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages Aug 28 09:58:06 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 09:58:06 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5 previous similar messages Aug 28 09:58:06 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 09:58:06 oss2 kernel: Lustre: 25843:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 10 previous similar messages Aug 28 09:58:13 oss2 kernel: LustreError: 10869:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810784649050 x1409779385570153/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169593 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:58:13 oss2 kernel: LustreError: 10869:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 4 previous similar messages Aug 28 09:59:23 oss2 kernel: Lustre: 18486:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 09:59:23 oss2 kernel: Lustre: 18486:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 10 previous similar messages Aug 28 09:59:23 oss2 kernel: LustreError: 18486:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81012f99c400 x1409779385583986/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169663 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 09:59:23 oss2 kernel: LustreError: 18486:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9 previous similar messages Aug 28 10:00:40 oss2 kernel: Lustre: 13236:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:00:40 oss2 kernel: Lustre: 13236:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 21 previous similar messages Aug 28 10:01:36 oss2 kernel: LustreError: 10861:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81009f9a4400 x1409779385607955/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346169796 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:01:36 oss2 kernel: LustreError: 10861:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 18 previous similar messages Aug 28 10:01:57 oss2 kernel: Lustre: 13259:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:01:57 oss2 kernel: Lustre: 13259:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 21 previous similar messages Aug 28 10:05:41 oss2 kernel: Lustre: 10815:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:05:41 oss2 kernel: Lustre: 10815:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 42 previous similar messages Aug 28 10:05:55 oss2 kernel: LustreError: 5170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b2596ec50 x1409779385656230/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346170055 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:05:55 oss2 kernel: LustreError: 5170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 36 previous similar messages Aug 28 10:06:58 oss2 kernel: Lustre: 5173:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:06:58 oss2 kernel: Lustre: 5173:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 42 previous similar messages Aug 28 10:14:33 oss2 kernel: LustreError: 13170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81070eb1e400 x1409779385752026/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346170573 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:14:33 oss2 kernel: LustreError: 13170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 73 previous similar messages Aug 28 10:15:43 oss2 kernel: Lustre: 5206:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:15:43 oss2 kernel: Lustre: 5206:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 10:17:00 oss2 kernel: Lustre: 5162:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:17:00 oss2 kernel: Lustre: 5162:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 10:24:35 oss2 kernel: LustreError: 18438:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8101c6e47c50 x1409779385868892/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346171175 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:24:35 oss2 kernel: LustreError: 18438:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 10:25:45 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:25:45 oss2 kernel: Lustre: 13246:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 10:27:02 oss2 kernel: Lustre: 18432:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:27:02 oss2 kernel: Lustre: 18432:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 10:34:37 oss2 kernel: LustreError: 15504:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8105e40b1c00 x1409779385886893/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346171777 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:34:37 oss2 kernel: LustreError: 15504:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 10:35:47 oss2 kernel: Lustre: 13203:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:35:47 oss2 kernel: Lustre: 13203:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 10:37:04 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:37:04 oss2 kernel: Lustre: 24878:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 10:44:39 oss2 kernel: LustreError: 15538:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c05c7ac00 x1409779385887637/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346172379 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:44:39 oss2 kernel: LustreError: 15538:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 10:45:49 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:45:49 oss2 kernel: Lustre: 13195:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 10:47:06 oss2 kernel: Lustre: 5186:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:47:06 oss2 kernel: Lustre: 5186:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 10:54:41 oss2 kernel: LustreError: 13221:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8102a01d6400 x1409779385888380/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346172981 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 10:54:41 oss2 kernel: LustreError: 13221:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 10:55:51 oss2 kernel: Lustre: 15515:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 10:55:51 oss2 kernel: Lustre: 15515:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 10:57:08 oss2 kernel: Lustre: 18414:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 10:57:08 oss2 kernel: Lustre: 18414:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:04:43 oss2 kernel: LustreError: 5205:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81047f6dd800 x1409779385889090/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346173583 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:04:43 oss2 kernel: LustreError: 5205:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:05:53 oss2 kernel: Lustre: 15507:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:05:53 oss2 kernel: Lustre: 15507:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:07:10 oss2 kernel: Lustre: 24893:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:07:10 oss2 kernel: Lustre: 24893:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:14:45 oss2 kernel: LustreError: 15555:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81032ef50800 x1409779385889800/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346174185 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:14:45 oss2 kernel: LustreError: 15555:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:15:55 oss2 kernel: Lustre: 18447:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:15:55 oss2 kernel: Lustre: 18447:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:17:12 oss2 kernel: Lustre: 15549:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:17:12 oss2 kernel: Lustre: 15549:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:24:47 oss2 kernel: LustreError: 16658:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810b3f6d0000 x1409779385890510/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346174787 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:24:47 oss2 kernel: LustreError: 16658:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:25:57 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:25:57 oss2 kernel: Lustre: 13210:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:27:14 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:27:14 oss2 kernel: Lustre: 13278:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:34:49 oss2 kernel: LustreError: 5176:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8104ba0bf000 x1409779385891246/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346175389 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:34:49 oss2 kernel: LustreError: 5176:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:35:59 oss2 kernel: Lustre: 5186:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:35:59 oss2 kernel: Lustre: 5186:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:37:16 oss2 kernel: Lustre: 24879:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:37:16 oss2 kernel: Lustre: 24879:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:44:51 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81073ccb9000 x1409779385891956/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346175991 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:44:51 oss2 kernel: LustreError: 13227:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:46:01 oss2 kernel: Lustre: 15552:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:46:01 oss2 kernel: Lustre: 15552:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:47:18 oss2 kernel: Lustre: 25839:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:47:18 oss2 kernel: Lustre: 25839:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 11:54:53 oss2 kernel: LustreError: 15538:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81061c8dac00 x1409779385892666/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346176593 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 11:54:53 oss2 kernel: LustreError: 15538:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 11:56:03 oss2 kernel: Lustre: 5239:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 11:56:03 oss2 kernel: Lustre: 5239:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 11:57:20 oss2 kernel: Lustre: 5164:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 11:57:20 oss2 kernel: Lustre: 5164:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:04:55 oss2 kernel: LustreError: 10808:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810a1fcad400 x1409779385893376/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346177195 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:04:55 oss2 kernel: LustreError: 10808:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:06:05 oss2 kernel: Lustre: 25834:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:06:05 oss2 kernel: Lustre: 25834:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:07:22 oss2 kernel: Lustre: 18412:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:07:22 oss2 kernel: Lustre: 18412:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:14:57 oss2 kernel: LustreError: 13278:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8109a33b2400 x1409779385894086/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346177797 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:14:57 oss2 kernel: LustreError: 13278:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:16:07 oss2 kernel: Lustre: 15522:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:16:07 oss2 kernel: Lustre: 15522:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:17:24 oss2 kernel: Lustre: 5174:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:17:24 oss2 kernel: Lustre: 5174:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:24:59 oss2 kernel: LustreError: 13271:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810147052800 x1409779385898218/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346178399 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:24:59 oss2 kernel: LustreError: 13271:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:26:09 oss2 kernel: Lustre: 13251:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:26:09 oss2 kernel: Lustre: 13251:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:27:26 oss2 kernel: Lustre: 13207:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:27:26 oss2 kernel: Lustre: 13207:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:35:01 oss2 kernel: LustreError: 13198:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810a794f8400 x1409779386043839/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346179001 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:35:01 oss2 kernel: LustreError: 13198:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:36:11 oss2 kernel: Lustre: 15544:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:36:11 oss2 kernel: Lustre: 15544:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:37:28 oss2 kernel: Lustre: 10870:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:37:28 oss2 kernel: Lustre: 10870:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:45:03 oss2 kernel: LustreError: 13267:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81065e4d1400 x1409779386257856/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346179603 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:45:03 oss2 kernel: LustreError: 13267:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:46:13 oss2 kernel: Lustre: 15501:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:46:13 oss2 kernel: Lustre: 15501:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:47:30 oss2 kernel: Lustre: 13211:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:47:30 oss2 kernel: Lustre: 13211:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 12:55:05 oss2 kernel: LustreError: 10807:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810c3dafb400 x1409779386455183/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346180205 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 12:55:05 oss2 kernel: LustreError: 10807:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 12:56:15 oss2 kernel: Lustre: 15532:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 12:56:15 oss2 kernel: Lustre: 15532:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 12:57:32 oss2 kernel: Lustre: 15509:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 12:57:32 oss2 kernel: Lustre: 15509:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:05:07 oss2 kernel: LustreError: 5201:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81047f6dd000 x1409779386641797/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346180807 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:05:07 oss2 kernel: LustreError: 5201:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:06:17 oss2 kernel: Lustre: 18430:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:06:17 oss2 kernel: Lustre: 18430:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:07:34 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:07:34 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:15:09 oss2 kernel: LustreError: 15525:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8107a4e9e400 x1409779386825295/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346181409 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:15:09 oss2 kernel: LustreError: 15525:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:16:19 oss2 kernel: Lustre: 13282:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:16:19 oss2 kernel: Lustre: 13282:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:17:36 oss2 kernel: Lustre: 18478:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:17:36 oss2 kernel: Lustre: 18478:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:25:11 oss2 kernel: LustreError: 18433:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8102c532d400 x1409779387009441/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346182011 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:25:11 oss2 kernel: LustreError: 18433:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:26:21 oss2 kernel: Lustre: 10829:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:26:21 oss2 kernel: Lustre: 10829:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:27:38 oss2 kernel: Lustre: 13187:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:27:38 oss2 kernel: Lustre: 13187:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:35:13 oss2 kernel: LustreError: 18428:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810895cf5000 x1409779387189094/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346182613 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:35:13 oss2 kernel: LustreError: 18428:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:36:23 oss2 kernel: Lustre: 15554:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:36:23 oss2 kernel: Lustre: 15554:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:37:40 oss2 kernel: Lustre: 5230:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:37:40 oss2 kernel: Lustre: 5230:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:45:15 oss2 kernel: LustreError: 18413:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81042f430c00 x1409779387365589/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346183215 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:45:15 oss2 kernel: LustreError: 18413:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:46:25 oss2 kernel: Lustre: 10831:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:46:25 oss2 kernel: Lustre: 10831:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:47:42 oss2 kernel: Lustre: 5231:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:47:42 oss2 kernel: Lustre: 5231:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 13:55:17 oss2 kernel: LustreError: 24879:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81044c275000 x1409779387438664/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346183817 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 13:55:17 oss2 kernel: LustreError: 24879:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 13:56:27 oss2 kernel: Lustre: 13239:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 13:56:27 oss2 kernel: Lustre: 13239:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 13:57:11 oss2 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.2.42@o2ib ns: filter-lustre-OST0011_UUID lock: ffff810beebf4000/0xf1f3ed1f8ee6337a lrc: 3/0,0 mode: PR/PR res: 4968246/0 rrc: 10 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0x654c64146823dc95 expref: 1045 pid: 10851 timeout 22172687605 Aug 28 13:57:44 oss2 kernel: Lustre: 25838:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 13:57:44 oss2 kernel: Lustre: 25838:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:05:19 oss2 kernel: LustreError: 15552:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81008c4b6c00 x1409779387522877/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346184419 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:05:19 oss2 kernel: LustreError: 15552:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 86 previous similar messages Aug 28 14:06:29 oss2 kernel: Lustre: 13186:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:06:29 oss2 kernel: Lustre: 13186:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:07:46 oss2 kernel: Lustre: 18414:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:07:46 oss2 kernel: Lustre: 18414:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:15:21 oss2 kernel: LustreError: 25857:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810abd0de800 x1409779387706631/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346185021 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:15:21 oss2 kernel: LustreError: 25857:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 14:16:31 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:16:31 oss2 kernel: Lustre: 13190:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:17:48 oss2 kernel: Lustre: 5253:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:17:48 oss2 kernel: Lustre: 5253:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:25:23 oss2 kernel: LustreError: 10874:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810bc4265400 x1409779387890904/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346185623 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:25:23 oss2 kernel: LustreError: 10874:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 14:26:33 oss2 kernel: Lustre: 24881:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:26:33 oss2 kernel: Lustre: 24881:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:27:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:27:50 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:35:25 oss2 kernel: LustreError: 10827:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810abd0de800 x1409779388073636/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346186225 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:35:25 oss2 kernel: LustreError: 10827:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 14:36:35 oss2 kernel: Lustre: 18473:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:36:35 oss2 kernel: Lustre: 18473:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:37:52 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:37:52 oss2 kernel: Lustre: 24892:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:45:27 oss2 kernel: LustreError: 5227:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810a00950800 x1409779388253683/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346186827 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:45:27 oss2 kernel: LustreError: 5227:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 14:46:37 oss2 kernel: Lustre: 5244:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:46:37 oss2 kernel: Lustre: 5244:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:47:54 oss2 kernel: Lustre: 13202:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:47:54 oss2 kernel: Lustre: 13202:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 14:55:29 oss2 kernel: LustreError: 18420:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81086d9b7c00 x1409779388392279/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346187429 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 14:55:29 oss2 kernel: LustreError: 18420:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 14:56:39 oss2 kernel: Lustre: 15544:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 14:56:39 oss2 kernel: Lustre: 15544:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 14:57:56 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 14:57:56 oss2 kernel: Lustre: 18411:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:05:31 oss2 kernel: LustreError: 10831:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8106b0cf5850 x1409779388478152/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346188031 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:05:31 oss2 kernel: LustreError: 10831:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:06:41 oss2 kernel: Lustre: 15502:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:06:41 oss2 kernel: Lustre: 15502:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:07:58 oss2 kernel: Lustre: 10842:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:07:58 oss2 kernel: Lustre: 10842:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:15:33 oss2 kernel: LustreError: 5216:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8104269d4000 x1409779388541011/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346188633 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:15:33 oss2 kernel: LustreError: 5216:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:16:43 oss2 kernel: Lustre: 13241:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:16:43 oss2 kernel: Lustre: 13241:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:18:00 oss2 kernel: Lustre: 5216:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:18:00 oss2 kernel: Lustre: 5216:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:25:35 oss2 kernel: LustreError: 15502:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8108e587c800 x1409779388543066/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346189235 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:25:35 oss2 kernel: LustreError: 15502:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:26:45 oss2 kernel: Lustre: 5157:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:26:45 oss2 kernel: Lustre: 5157:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:28:02 oss2 kernel: Lustre: 18480:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:28:02 oss2 kernel: Lustre: 18480:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:35:37 oss2 kernel: LustreError: 13218:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8103f1cd3800 x1409779388545121/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346189837 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:35:37 oss2 kernel: LustreError: 13218:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:36:47 oss2 kernel: Lustre: 18460:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:36:47 oss2 kernel: Lustre: 18460:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:38:04 oss2 kernel: Lustre: 10807:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:38:04 oss2 kernel: Lustre: 10807:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:45:39 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81075706c800 x1409779388547201/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346190439 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:45:39 oss2 kernel: LustreError: 13179:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:46:49 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:46:49 oss2 kernel: Lustre: 13180:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:48:06 oss2 kernel: Lustre: 13247:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:48:06 oss2 kernel: Lustre: 13247:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 15:55:41 oss2 kernel: LustreError: 13261:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff8109a33b2000 x1409779388549256/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346191041 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 15:55:41 oss2 kernel: LustreError: 13261:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 15:56:51 oss2 kernel: Lustre: 24890:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 15:56:51 oss2 kernel: Lustre: 24890:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 15:58:08 oss2 kernel: Lustre: 13290:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 15:58:08 oss2 kernel: Lustre: 13290:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 16:05:43 oss2 kernel: LustreError: 24883:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff81021775ec50 x1409779388551317/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346191643 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 16:05:43 oss2 kernel: LustreError: 24883:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 16:06:53 oss2 kernel: Lustre: 5250:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 16:06:53 oss2 kernel: Lustre: 5250:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 16:08:10 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 16:08:10 oss2 kernel: Lustre: 5191:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages Aug 28 16:15:45 oss2 kernel: LustreError: 24876:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810938975400 x1409779388553378/t0 o8->cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@NET_0x50000c0a80269_UUID:0/0 lens 368/264 e 0 to 0 dl 1346192245 ref 1 fl Interpret:/0/0 rc -16/0 Aug 28 16:15:45 oss2 kernel: LustreError: 24876:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 85 previous similar messages Aug 28 16:16:55 oss2 kernel: Lustre: 15541:0:(ldlm_lib.c:874:target_handle_connect()) lustre-OST0009: refuse reconnection from cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3@192.168.2.105@o2ib to 0xffff81061dafc200; still busy with 1 active RPCs Aug 28 16:16:55 oss2 kernel: Lustre: 15541:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 85 previous similar messages Aug 28 16:18:12 oss2 kernel: Lustre: 13285:0:(ldlm_lib.c:574:target_handle_reconnect()) lustre-OST0009: cbd7b84c-ff9b-9d3a-f66f-455fdf6022f3 reconnecting Aug 28 16:18:12 oss2 kernel: Lustre: 13285:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 85 previous similar messages