Linux version 2.6.18-238.9.1.el5_lustre.gdea1dfa (jenkins@rhel5-32-build.lab.whamcloud.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Tue Jun 7 12:45:44 PDT 2011 BIOS-provided physical RAM map: BIOS-e820: 0000000000010000 - 000000000009cc00 (usable) BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 00000000bf760000 (usable) BIOS-e820: 00000000bf76e000 - 00000000bf770000 type 9 BIOS-e820: 00000000bf770000 - 00000000bf77e000 (ACPI data) BIOS-e820: 00000000bf77e000 - 00000000bf7d0000 (ACPI NVS) BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved) BIOS-e820: 00000000bf7ec000 - 00000000c0000000 (reserved) BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 0000000640000000 (usable) Warning only 4GB will be used. Use a PAE enabled kernel. 3200MB HIGHMEM available. 896MB LOWMEM available. found SMP MP-table at 000ff780 Memory for crash kernel (0x0 to 0x0) notwithin permissible range disabling kdump Using x86 segment limits to approximate NX protection On node 0 totalpages: 1048576 DMA zone: 4096 pages, LIFO batch:0 Normal zone: 225280 pages, LIFO batch:31 HighMem zone: 819200 pages, LIFO batch:31 DMI present. Using APIC driver default ACPI: RSDP (v002 ACPIAM ) @ 0x000f9fb0 ACPI: XSDT (v001 SMCI 0x20101005 MSFT 0x00000097) @ 0xbf770100 ACPI: FADT (v003 100510 FACP1516 0x20101005 MSFT 0x00000097) @ 0xbf770290 ACPI: MADT (v001 100510 APIC1516 0x20101005 MSFT 0x00000097) @ 0xbf770390 ACPI: MCFG (v001 100510 OEMMCFG 0x20101005 MSFT 0x00000097) @ 0xbf7704b0 ACPI: SLIT (v001 100510 OEMSLIT 0x20101005 MSFT 0x00000097) @ 0xbf7704f0 ACPI: OEMB (v001 100510 OEMB1516 0x20101005 MSFT 0x00000097) @ 0xbf77e040 ACPI: SRAT (v001 100510 OEMSRAT 0x00000001 INTL 0x00000001) @ 0xbf77a6a0 ACPI: HPET (v001 100510 OEMHPET 0x20101005 MSFT 0x00000097) @ 0xbf77a8f0 ACPI: DMAR (v001 AMI OEMDMAR 0x00000001 MSFT 0x00000097) @ 0xbf77e0d0 ACPI: SSDT (v001 DpgPmm CpuPm 0x00000012 INTL 0x20051117) @ 0xbf784270 ACPI: EINJ (v001 AMIER AMI_EINJ 0x20101005 MSFT 0x00000097) @ 0xbf77a930 ACPI: BERT (v001 AMIER AMI_BERT 0x20101005 MSFT 0x00000097) @ 0xbf77aac0 ACPI: ERST (v001 AMIER AMI_ERST 0x20101005 MSFT 0x00000097) @ 0xbf77aaf0 ACPI: HEST (v001 AMIER ABC_HEST 0x20101005 MSFT 0x00000097) @ 0xbf77aca0 ACPI: DSDT (v001 30007 30007000 0x00000000 INTL 0x20051117) @ 0x00000000 ACPI: PM-Timer IO Port: 0x808 ACPI: Local APIC address 0xfee00000 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Processor #0 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) Processor #2 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) Processor #4 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled) Processor #16 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled) Processor #18 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled) Processor #20 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x20] enabled) Processor #32 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x22] enabled) Processor #34 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x24] enabled) Processor #36 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x30] enabled) Processor #48 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) Processor #50 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x34] enabled) Processor #52 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x01] enabled) Processor #1 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x03] enabled) Processor #3 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x05] enabled) Processor #5 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x10] lapic_id[0x11] enabled) Processor #17 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x11] lapic_id[0x13] enabled) Processor #19 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x12] lapic_id[0x15] enabled) Processor #21 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x13] lapic_id[0x21] enabled) Processor #33 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x14] lapic_id[0x23] enabled) Processor #35 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] enabled) Processor #37 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x16] lapic_id[0x31] enabled) Processor #49 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x17] lapic_id[0x33] enabled) Processor #51 6:12 APIC version 21 ACPI: LAPIC (acpi_id[0x18] lapic_id[0x35] enabled) Processor #53 6:12 APIC version 21 ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) Overriding APIC driver with bigsmp ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23 ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24]) IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: IRQ0 used by override. ACPI: IRQ2 used by override. ACPI: IRQ9 used by override. Enabling APIC mode: Physflat. Using 2 I/O APICs ACPI: HPET id: 0x8086a301 base: 0xfed00000 Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at c2000000 (gap: c0000000:20000000) Detected 2666.923 MHz processor. Built 1 zonelists. Total pages: 1048576 Kernel command line: ro root=LABEL=/ console=ttyS0,115200 mapped APIC to ffffd000 (fee00000) mapped IOAPIC to ffffc000 (fec00000) mapped IOAPIC to ffffb000 (fec8a000) Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 PID hash table entries: 4096 (order: 12, 16384 bytes) Console: colour VGA+ 80x25 Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Memory: 3094400k/4194304k available (2193k kernel code, 41148k reserved, 913k data, 228k init, 2219392k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. hpet0: at MMIO 0xfed00000 (virtual 0xf8800000), IRQs 2, 8, 0, 0 hpet0: 4 64-bit timers, 14318180 Hz Using HPET for base-timer Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.84 BogoMIPS (lpj=2666923) Security Framework v1.0.0 initialized SELinux: Initializing. SELinux: Starting in permissive mode selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 512 CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. using mwait in idle threads. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. Checking 'hlt' instruction... OK. SMP alternatives: switching to UP code ACPI: Core revision 20060707 CPU0: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 Leaving ESR disabled. SMP alternatives: switching to SMP code Booting processor 1/2 eip 11000 Initializing CPU#1 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666742) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 1 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#1. CPU1: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 2/4 eip 11000 Initializing CPU#2 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666744) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 2 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#2. CPU2: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 3/16 eip 11000 Initializing CPU#3 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.49 BogoMIPS (lpj=2666746) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 8 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#3. CPU3: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 4/18 eip 11000 Initializing CPU#4 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666743) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 9 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#4. CPU4: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 5/20 eip 11000 Initializing CPU#5 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666743) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 10 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#5. CPU5: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 6/32 eip 11000 Initializing CPU#6 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666763) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 0 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#6. CPU6: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 7/34 eip 11000 Initializing CPU#7 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.53 BogoMIPS (lpj=2666766) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 1 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#7. CPU7: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 8/36 eip 11000 Initializing CPU#8 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666762) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#8. CPU8: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 9/48 eip 11000 Initializing CPU#9 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.51 BogoMIPS (lpj=2666758) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 8 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#9. CPU9: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 10/50 eip 11000 Initializing CPU#10 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.51 BogoMIPS (lpj=2666759) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 9 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#10. CPU10: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 11/52 eip 11000 Initializing CPU#11 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.51 BogoMIPS (lpj=2666755) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 10 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#11. CPU11: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 12/1 eip 11000 Initializing CPU#12 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.47 BogoMIPS (lpj=2666737) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#12. CPU12: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 13/3 eip 11000 Initializing CPU#13 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.49 BogoMIPS (lpj=2666746) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 1 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#13. CPU13: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 14/5 eip 11000 Initializing CPU#14 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666744) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 2 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#14. CPU14: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 15/17 eip 11000 Initializing CPU#15 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.47 BogoMIPS (lpj=2666738) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 8 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#15. CPU15: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 16/19 eip 11000 Initializing CPU#16 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.50 BogoMIPS (lpj=2666750) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 9 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#16. CPU16: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 17/21 eip 11000 Initializing CPU#17 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.48 BogoMIPS (lpj=2666744) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 0 CPU: Processor Core ID: 10 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#17. CPU17: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 18/33 eip 11000 Initializing CPU#18 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666763) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 0 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#18. CPU18: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 19/35 eip 11000 Initializing CPU#19 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666764) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 1 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#19. CPU19: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 20/37 eip 11000 Initializing CPU#20 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666762) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#20. CPU20: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 21/49 eip 11000 Initializing CPU#21 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.52 BogoMIPS (lpj=2666764) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 8 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#21. CPU21: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 22/51 eip 11000 Initializing CPU#22 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.53 BogoMIPS (lpj=2666765) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 9 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#22. CPU22: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 SMP alternatives: switching to SMP code Booting processor 23/53 eip 11000 Initializing CPU#23 Leaving ESR disabled. Calibrating delay using timer specific routine.. 5333.51 BogoMIPS (lpj=2666758) CPU: After generic identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 CPU: After vendor identify, caps: bfebfbff 2c100000 00000000 02010000 009ee3fd 00000000 00000001 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 256K CPU: L3 cache: 12288K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 10 CPU: After all inits, caps: bfebf3ff 2c100000 00000000 13010940 009ee3fd 00000000 00000001 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#23. CPU23: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping 02 Total of 24 processors activated (128004.47 BogoMIPS). ENABLING IO-APIC IRQs ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1 Using local APIC timer interrupts. checking TSC synchronization across 24 CPUs: passed. Brought up 24 CPUs sizeof(vma)=84 bytes sizeof(page)=32 bytes sizeof(inode)=340 bytes sizeof(dentry)=136 bytes sizeof(ext3inode)=492 bytes sizeof(buffer_head)=52 bytes sizeof(skbuff)=176 bytes migration_cost=11,103,1345 checking if image is initramfs... it is Freeing initrd memory: 2542k freed NET: Registered protocol family 16 ACPI: bus type pci registered PCI: Using MMCONFIG Setting up standard PCI resources ACPI: Interpreter enabled ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. ACPI: PCI Root Bridge [PCI0] (0000:00) PCI: Transparent bridge - 0000:00:1e.0 ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE1._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE2._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE7._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE3._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.NPE5._PRT] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11 12 14 *15) ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15) ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 12 14 15) ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15) ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 *14 15) ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15) Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI init pnp: PnP ACPI: found 15 devices usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: Using ACPI for IRQ routing PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default pnp: 00:01: iomem range 0xfbf00000-0xfbffffff has been reserved pnp: 00:01: iomem range 0xfc000000-0xfcffffff has been reserved pnp: 00:01: iomem range 0xfd000000-0xfdffffff has been reserved pnp: 00:01: iomem range 0xfe000000-0xfebfffff has been reserved pnp: 00:08: ioport range 0x164e-0x164f has been reserved pnp: 00:08: ioport range 0xa00-0xa0f has been reserved pnp: 00:09: iomem range 0xfed1c000-0xfed1ffff has been reserved pnp: 00:09: iomem range 0xfed20000-0xfed3ffff has been reserved pnp: 00:09: iomem range 0xfed40000-0xfed8ffff has been reserved pnp: 00:0b: iomem range 0xfec00000-0xfec00fff has been reserved pnp: 00:0b: iomem range 0xfee00000-0xfee00fff could not be reserved pnp: 00:0c: ioport range 0xca2-0xca3 has been reserved pnp: 00:0d: iomem range 0xe0000000-0xefffffff could not be reserved pnp: 00:0e: iomem range 0x0-0x9ffff could not be reserved pnp: 00:0e: iomem range 0xc0000-0xcffff could not be reserved pnp: 00:0e: iomem range 0xe0000-0xfffff could not be reserved pnp: 00:0e: iomem range 0x100000-0xbfffffff could not be reserved PCI: Bridge: 0000:00:01.0 IO window: d000-0000 MEM window: fba00000-00000000 PREFETCH window: disabled. PCI: Bridge: 0000:00:02.0 IO window: e000-0000 MEM window: fbb00000-00000000 PREFETCH window: disabled. PCI: Bridge: 0000:00:03.0 IO window: disabled. MEM window: fbc00000-00000000 PREFETCH window 0x00000000f8000000-0x00000000f87fffff PCI: Bridge: 0000:00:05.0 IO window: disabled. MEM window: disabled. PREFETCH window: disabled. PCI: Bridge: 0000:00:07.0 IO window: disabled. MEM window: fbd00000-00000000 PREFETCH window 0x00000000f8800000-0x00000000f8ffffff PCI: Bridge: 0000:00:1e.0 IO window: disabled. MEM window: faf00000-00000000 PREFETCH window 0x00000000f9000000-0x00000000f9ffffff PCI: Setting latency timer of device 0000:00:01.0 to 64 PCI: Setting latency timer of device 0000:00:02.0 to 64 PCI: Setting latency timer of device 0000:00:03.0 to 64 PCI: Setting latency timer of device 0000:00:05.0 to 64 PCI: Setting latency timer of device 0000:00:07.0 to 64 PCI: Setting latency timer of device 0000:00:1e.0 to 64 NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 5, 131072 bytes) TCP established hash table entries: 131072 (order: 8, 1048576 bytes) TCP bind hash table entries: 65536 (order: 7, 524288 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered apm: BIOS not found. audit: initializing netlink socket (disabled) type=2000 audit(1307995726.369:1): initialized highmem bounce pool size: 64 pages Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) SELinux: Registering netfilter hooks Initializing Cryptographic API alg: No test for crc32c (crc32c-generic) ksign: Installing public key data Loading keyring io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered (default) io scheduler cfq registered Boot video device is 0000:06:01.0 PCI: Setting latency timer of device 0000:00:01.0 to 64 PCI: Setting latency timer of device 0000:00:02.0 to 64 PCI: Setting latency timer of device 0000:00:03.0 to 64 PCI: Setting latency timer of device 0000:00:05.0 to 64 PCI: Setting latency timer of device 0000:00:07.0 to 64 pci_hotplug: PCI Hot Plug PCI Core version: 0.5 ACPI (exconfig-0456): Dynamic SSDT Load - OemId [DpgPmm] OemTableId [ P001Ist] [20060707] ACPI (exconfig-0456): Dynamic SSDT Load - OemId [ PmRef] OemTableId [ P001Cst] [20060707] Monitor-Mwait will be used to enter C-2 state Monitor-Mwait will be used to enter C-3 state ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU4 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU5 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU6 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU7 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU8 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU9 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU10 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU11 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU12 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU13 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU14 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU15 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU16 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU17 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU18 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU19 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU20 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU21 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU22 (power states: C1[C1] C2[C2] C3[C3]) ACPI: CPU23 (power states: C1[C1] C2[C2] C3[C3]) Real Time Clock Driver v1.12ac hpet_resources: 0xfed00000 is busy Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A brd: module loaded Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Probing IDE interface ide0... Probing IDE interface ide1... ide-floppy driver 0.99.newide usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.6:USB HID core driver PNP: No PS/2 controller found. Probing ports directly. serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: bitmap version 4.39 TCP bic registered Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 Using IPI No-Shortcut mode ACPI: (supports<6>Time: tsc clocksource has been installed. S0 S1 S4 S5) Initalizing network drop monitor service Freeing unused kernel memory: 228k freed Write protecting the kernel read-only data: 414k ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 217 PCI: Setting latency timer of device 0000:00:1a.7 to 64 ehci_hcd 0000:00:1a.7: EHCI Host Controller ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:1a.7: debug port 1 PCI: cache line size of 32 is not supported by device 0000:00:1a.7 ehci_hcd 0000:00:1a.7: irq 217, io mem 0xfbed6000 ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 6 ports detected ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 225 PCI: Setting latency timer of device 0000:00:1d.7 to 64 ehci_hcd 0000:00:1d.7: EHCI Host Controller ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 ehci_hcd 0000:00:1d.7: debug port 1 PCI: cache line size of 32 is not supported by device 0000:00:1d.7 ehci_hcd 0000:00:1d.7: irq 225, io mem 0xfbed4000 ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 6 ports detected ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI) USB Universal Host Controller Interface driver v3.0 ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 233 PCI: Setting latency timer of device 0000:00:1a.0 to 64 uhci_hcd 0000:00:1a.0: UHCI Host Controller uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1a.0: irq 233, io base 0x0000b880 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 21 (level, low) -> IRQ 50 PCI: Setting latency timer of device 0000:00:1a.1 to 64 uhci_hcd 0000:00:1a.1: UHCI Host Controller uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 uhci_hcd 0000:00:1a.1: irq 50, io base 0x0000bc00 usb usb4: configuration #1 chosen from 1 choice hub 4-0:1.0: USB hub found hub 4-0:1.0: 2 ports detected ACPI: PCI Interrupt 0000:00:1a.2[D] -> GSI 19 (level, low) -> IRQ 58 PCI: Setting latency timer of device 0000:00:1a.2 to 64 uhci_hcd 0000:00:1a.2: UHCI Host Controller uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5 uhci_hcd 0000:00:1a.2: irq 58, io base 0x0000c000 usb usb5: configuration #1 chosen from 1 choice hub 5-0:1.0: USB hub found hub 5-0:1.0: 2 ports detected ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 225 PCI: Setting latency timer of device 0000:00:1d.0 to 64 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6 uhci_hcd 0000:00:1d.0: irq 225, io base 0x0000b400 usb usb6: configuration #1 chosen from 1 choice hub 6-0:1.0: USB hub found hub 6-0:1.0: 2 ports detected usb 4-1: new full speed USB device using uhci_hcd and address 2 ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 58 PCI: Setting latency timer of device 0000:00:1d.1 to 64 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7 uhci_hcd 0000:00:1d.1: irq 58, io base 0x0000b480 usb usb7: configuration #1 chosen from 1 choice hub 7-0:1.0: USB hub found hub 7-0:1.0: 2 ports detected usb 4-1: configuration #1 chosen from 1 choice ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 217 PCI: Setting latency timer of device 0000:00:1d.2 to 64 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8 uhci_hcd 0000:00:1d.2: irq 217, io base 0x0000b800 input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input0 input: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 input: American Megatrends Inc. Virtual Keyboard and Mouse as /class/input/input1 input: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1 usb usb8: configuration #1 chosen from 1 choice hub 8-0:1.0: USB hub found hub 8-0:1.0: 2 ports detected SCSI subsystem initialized libata version 3.00 loaded. ahci 0000:00:1f.2: version 3.0 ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 58 ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ems PCI: Setting latency timer of device 0000:00:1f.2 to 64 scsi0 : ahci scsi1 : ahci scsi2 : ahci scsi3 : ahci scsi4 : ahci scsi5 : ahci ata1: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa100 irq 66 ata2: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa180 irq 66 ata3: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa200 irq 66 ata4: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa280 irq 66 ata5: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa300 irq 66 ata6: SATA max UDMA/133 abar m2048@0xfbefa000 port 0xfbefa380 irq 66 ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata1.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133 ata1.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32) ata1.00: configured for UDMA/133 ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata2.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133 ata2.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32) ata2.00: configured for UDMA/133 ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata3.00: ATA-8: WDC WD2502ABYS-02B7A0, 02.03B03, max UDMA/133 ata3.00: 490350672 sectors, multi 0: LBA48 NCQ (depth 31/32) ata3.00: configured for UDMA/133 ata4: SATA link down (SStatus 0 SControl 300) ata5: SATA link down (SStatus 0 SControl 300) ata6: SATA link down (SStatus 0 SControl 300) Vendor: ATA Model: WDC WD2502ABYS-0 Rev: 02.0 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) sda: Write Protect is off sda: Mode Sense: 00 3a 00 00 SCSI device sda: drive cache: write back SCSI device sda: 490350672 512-byte hdwr sectors (251060 MB) sda: Write Protect is off sda: Mode Sense: 00 3a 00 00 SCSI device sda: drive cache: write back sda: sda1 sda2 sd 0:0:0:0: Attached scsi disk sda Vendor: ATA Model: WDC WD2502ABYS-0 Rev: 02.0 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sdb: 490350672 512-byte hdwr sectors (251060 MB) sdb: Write Protect is off sdb: Mode Sense: 00 3a 00 00 SCSI device sdb: drive cache: write back SCSI device sdb: 490350672 512-byte hdwr sectors (251060 MB) sdb: Write Protect is off sdb: Mode Sense: 00 3a 00 00 SCSI device sdb: drive cache: write back sdb: unknown partition table sd 1:0:0:0: Attached scsi disk sdb Vendor: ATA Model: WDC WD2502ABYS-0 Rev: 02.0 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sdc: 490350672 512-byte hdwr sectors (251060 MB) sdc: Write Protect is off sdc: Mode Sense: 00 3a 00 00 SCSI device sdc: drive cache: write back SCSI device sdc: 490350672 512-byte hdwr sectors (251060 MB) sdc: Write Protect is off sdc: Mode Sense: 00 3a 00 00 SCSI device sdc: drive cache: write back sdc: sdc1 sdc2 sdc3 sd 2:0:0:0: Attached scsi disk sdc device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com device-mapper: dm-raid45: initialized v0.2594l EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks type=1404 audit(1307995755.392:2): selinux=0 auid=4294967295 ses=4294967295 input: PC Speaker as /class/input/input2 ACPI: PCI Interrupt 0000:00:1f.3[C] -> GSI 18 (level, low) -> IRQ 217 mlx4_core: Mellanox ConnectX core driver v1.0 (April 4, 2008) mlx4_core: Initializing 0000:03:00.0 ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 24 (level, low) -> IRQ 74 PCI: Setting latency timer of device 0000:03:00.0 to 64 e1000e: Intel(R) PRO/1000 Network Driver - 1.2.7-k2 e1000e: Copyright (c) 1999 - 2010 Intel Corporation. sd 0:0:0:0: Attached scsi generic sg0 type 0 sd 1:0:0:0: Attached scsi generic sg1 type 0 sd 2:0:0:0: Attached scsi generic sg2 type 0 Requesting 25 MSIX vectors mlx4_core: Initializing 0000:05:00.0 ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 30 (level, low) -> IRQ 91 PCI: Setting latency timer of device 0000:05:00.0 to 64 Requesting 25 MSIX vectors ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 28 (level, low) -> IRQ 108 PCI: Setting latency timer of device 0000:01:00.0 to 64 e1000e 0000:01:00.0: Disabling ASPM L0s mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.4.2.2 (Nov 2009) mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008) eth0: (PCI Express:2.5GB/s:Width x1) 00:25:90:1f:aa:50 eth0: Intel(R) PRO/1000 Network Connection eth0: MAC: 3, PHY: 8, PBA No: 0101ff-0ff ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 29 (level, low) -> IRQ 140 PCI: Setting latency timer of device 0000:02:00.0 to 64 e1000e 0000:02:00.0: Disabling ASPM L0s eth1: (PCI Express:2.5GB/s:Width x1) 00:25:90:1f:aa:51 eth1: Intel(R) PRO/1000 Network Connection eth1: MAC: 3, PHY: 8, PBA No: 0101ff-0ff floppy0: no floppy controllers found work still pending lp: driver loaded but no devices found ACPI: Power Button (FF) [PWRF] ACPI: Power Button (CM) [PWRB] ACPI: Mapper loaded dell-wmi: No known WMI GUID found md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. device-mapper: multipath: version 1.0.6 loaded EXT3 FS on sda1, internal journal Adding 5156856k swap on /dev/sda2. Priority:-1 extents:1 across:5156856k IA-32 Microcode Update Driver: v1.14a NET: Registered protocol family 10 lo: Disabled Privacy Extensions IPv6 over IPv4 tunneling driver NET: Registered protocol family 27 NET: Registered protocol family 28 Registered RDS/iwarp transport Registered RDS/infiniband transport Loading iSCSI transport class v2.0-871. iscsi: registered transport (iser) 802.1Q VLAN Support v1.8 Ben Greear All bugs added by David S. Miller cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits. iscsi: registered transport (cxgb3i) Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010) Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010) iscsi: registered transport (bnx2i) iscsi: registered transport (tcp) iscsi: registered transport (be2iscsi) ADDRCONF(NETDEV_UP): eth0: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ADDRCONF(NETDEV_UP): eth1: link is not ready ADDRCONF(NETDEV_UP): ib0: link is not ready ib_srp: ASYNC event= 17 on device= mlx4_0 ib_srp: ASYNC event= 11 on device= mlx4_0 ib_srp: ASYNC event= 9 on device= mlx4_0 ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P001._PPC] (Node f7d4de8c), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P002._PPC] (Node f7d4de14), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P003._PPC] (Node f7d4dd9c), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P004._PPC] (Node f7d4dd24), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P005._PPC] (Node f7d4dcac), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P006._PPC] (Node f7d4dc34), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P007._PPC] (Node f7d4dbbc), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P008._PPC] (Node f7d4db44), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P009._PPC] (Node f7d4dacc), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P010._PPC] (Node f7d4da54), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P011._PPC] (Node f7d4dfe0), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P012._PPC] (Node f7d4df68), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P013._PPC] (Node f7d4def0), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P014._PPC] (Node f74a8748), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P015._PPC] (Node f74a86d0), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P016._PPC] (Node f74a8658), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P017._PPC] (Node f74a85e0), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P018._PPC] (Node f74a8568), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P019._PPC] (Node f74a84f0), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P020._PPC] (Node f74a8478), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P021._PPC] (Node f74a8400), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P022._PPC] (Node f74a8388), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P023._PPC] (Node f74a8310), AE_NOT_FOUND ACPI Error (psargs-0355): [PSTE] Namespace lookup failure, AE_NOT_FOUND ACPI Error (psparse-0537): Method parse/execution failed [\_PR_.P024._PPC] (Node f74a8bf8), AE_NOT_FOUND Bluetooth: Core ver 2.10 NET: Registered protocol family 31 Bluetooth: HCI device and connection manager initialized Bluetooth: HCI socket layer initialized Bluetooth: L2CAP ver 2.8 Bluetooth: L2CAP socket layer initialized Bluetooth: RFCOMM socket layer initialized Bluetooth: RFCOMM TTY layer initialized Bluetooth: RFCOMM ver 1.8 eth0: no IPv6 routers present Bluetooth: HIDP (Human Interface Emulation) ver 1.1 ib0: no IPv6 routers present init dynlocks cache ldiskfs created from ext4-2.6-rhel5 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc3): mounted filesystem with ordered data mode Lustre: OBD class driver, http://www.lustre.org/ Lustre: Lustre Version: 2.0.62 Lustre: Build Version: jenkins-gdea1dfa-PRISTINE-2.6.18-238.9.1.el5_lustre.gdea1dfa Lustre: Lustre LU module (f9110820). Lustre: Register global MR array, MR size: 0xffffffffffffffff, array size: 1 Lustre: Added LNI 192.168.4.128@o2ib [8/64/0/180] Lustre: Lustre OSC module (f99c84c0). Lustre: Lustre LOV module (f9b09780). Lustre: Lustre client module (f9befb80). LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: 7030:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 90000: select flavor null Lustre: 7627:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from d4b56168-b6b7-9abd-95b7-8869d7bbf4c8@0@lo t0 exp 00000000 cur 1308021332 last 0 Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: MGS: Regenerating lustre-MDTffff log by user request. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: new disk, initializing Lustre: 7641:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 7830:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OSTffff log by user request. Lustre: lustre-OST0000: new disk, initializing Lustre: 8047:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled LustreError: 7941:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 8115:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 8115:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode Lustre: lustre-OST0001: new disk, initializing Lustre: 8289:0:(filter.c:1238:filter_prep_groups()) lustre-OST0001: initialize groups [0,0] Lustre: lustre-OST0001: Now serving lustre-OST0001 on /dev/sdc2 with recovery enabled LustreError: 8227:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0001 has no notify handler LDISKFS-fs (sdc3): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OSTffff log by user request. Lustre: Skipped 1 previous similar message Lustre: 8542:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 8542:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 3 previous similar messages Lustre: 7627:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from 8b2fff39-e741-63e8-58a7-140a8369f57f@192.168.4.15@o2ib t0 exp 00000000 cur 1308021335 last 0 Lustre: 7627:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 7627:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib LustreError: 7780:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-11) req@cb8e5000 x1371586166784010/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1308021356 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1 Lustre: 8587:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 8587:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-osc-MDT0000->192.168.4.128@o2ib netid 90000: select flavor null Lustre: 8029:0:(ldlm_lib.c:871:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@0@lo t0 exp 00000000 cur 1308021337 last 0 Lustre: 8028:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab80006802e0fa); group 0 Lustre: lustre-OST0001: received MDS connection from 0@lo Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: 8029:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 2 previous similar messages Lustre: 7780:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from d94eb03c-f960-e5f0-c46d-4cf0c55bff23@192.168.4.15@o2ib t0 exp 00000000 cur 1308021341 last 0 Lustre: 7780:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 7780:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Lustre: 8029:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab80006802e147); group 0 Lustre: 8029:0:(filter.c:2846:filter_connect()) Skipped 4 previous similar messages Lustre: 8029:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab80006802e19b); group 0 Lustre: 8029:0:(filter.c:2846:filter_connect()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: 7780:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Jun 14 03:15:53 PDT 2011 Lustre: DEBUG MARKER: excepting tests: 61d 33a 33b Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: 7780:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 7780:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Lustre: 9730:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 9730:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =============================================================== 03:15:57 (1308046557) LustreError: 10086:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: 10191:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 10191:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775174745 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308021367] [real_sent 1308021367] [current 1308021374] [deadline 7s] [delay 0s] req@dd702800 x1371559775174745/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308021374 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775174747 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308021374] [real_sent 1308021374] [current 1308021380] [deadline 6s] [delay 0s] req@d5239400 x1371559775174747/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308021380 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: 10304:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s Lustre: 10361:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from d4b56168-b6b7-9abd-95b7-8869d7bbf4c8@0@lo t0 exp 00000000 cur 1308021380 last 0 Lustre: 10361:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 8 previous similar messages Lustre: 10361:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->192.168.4.128@o2ib netid 90000: select flavor null Lustre: 10361:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 8 previous similar messages Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 10368:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 4294967298 LustreError: 10372:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 10372 Lustre: 10368:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 10368:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 10368:0:(mds_lov.c:1003:mds_notify()) Skipped 2 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8028:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab80006802e31c); group 0 Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0001_UUID Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 2 previous similar messages Lustre: 10427:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 10427:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: 10374:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 4294967299, current: 4294967299, replaying Lustre: 10374:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from 993b6001-16bd-8f2c-6a64-be226887a26f@192.168.4.18@o2ib recovering/t4294967300 exp e789c200 cur 1308021390 last 1308021380 Lustre: 10374:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 6 previous similar messages Lustre: 10374:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a80412_UUID netid 50000: select flavor null Lustre: 10374:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 9 previous similar messages Lustre: 10374:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 4294967299, current: 4294967300, replaying Lustre: 10374:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 1 previous similar message Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: 10372:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 10372:0:(mds_lov.c:1023:mds_notify()) Skipped 2 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xd16d2200 -> 0xe01ab800 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) changing the import cf6f5800 - d807ac00 Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: Skipped 2 previous similar messages Lustre: Skipped 4 previous similar messages Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================= 03:16:32 (1308046592) Lustre: Failing over lustre-OST0000 Lustre: Skipped 5 previous similar messages Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Lustre: server umount lustre-OST0000 complete LustreError: 8025:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@c899c400 x1371559775174794/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021401 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: 8029:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@f5be5400 x1371586166784089/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021404 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 8029:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 1 previous similar message LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 10825:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-OST0000, 3 recoverable clients, last_transno 0 LustreError: 10827:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-OST0000: started recovery thread pid 10827 Lustre: 10825:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 10825:0:(filter.c:1238:filter_prep_groups()) Skipped 1 previous similar message Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 10768:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 10768:0:(obd_class.h:1593:obd_notify()) Skipped 1 previous similar message Lustre: 10866:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 10866:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s Lustre: 8025:0:(ldlm_lib.c:871:target_handle_connect()) lustre-OST0000: connection from d94eb03c-f960-e5f0-c46d-4cf0c55bff23@192.168.4.15@o2ib recovering/t0 exp f3691c00 cur 1308021408 last 1308021403 Lustre: 8025:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 1 previous similar message Lustre: 8025:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 8025:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 1 previous similar message Lustre: lustre-OST0000: sending delayed replies to recovered clients Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients =========================================== 03:16:52 (1308046612) LustreError: 11119:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: 11224:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 11224:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 1 previous similar message Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 11399:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 8589934653 LustreError: 11404:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 11404 Lustre: 11399:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 11399:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 11399:0:(mds_lov.c:1003:mds_notify()) Skipped 2 previous similar messages Lustre: 8026:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8026:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab80006802ed10); group 0 Lustre: 8026:0:(filter.c:2846:filter_connect()) Skipped 2 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 2 previous similar messages Lustre: 11462:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 11462:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message LustreError: 11406:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 0 clients in recovery for 60s LustreError: 11406:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-16) req@cc391800 x1371586166784225/t0(0) o-1->@:0/0 lens 368/264 e 0 to 0 dl 1308021446 ref 1 fl Interpret:/ffffffff/ffffffff rc -16/-1 LustreError: 11406:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 3 previous similar messages LustreError: 11396:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 11406:0:(mdt_handler.c:2815:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 11406:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@df3d1800 x1371586212921451/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021455 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 Lustre: 11405:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 8589934654, current: 8589934654, replaying Lustre: 11405:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 1 previous similar message LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 80s LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 75s Lustre: 11405:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib recovering/t0 exp 00000000 cur 1308021441 last 0 Lustre: 11405:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 11 previous similar messages LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 70s LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-16) req@f7608c00 x1371586166784232/t0(0) o-1->@:0/0 lens 368/264 e 0 to 0 dl 1308021461 ref 1 fl Interpret:/ffffffff/ffffffff rc -16/-1 LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 3 previous similar messages LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 65s LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 55s LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 1 previous similar message LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-16) req@f7bf2400 x1371586166784240/t0(0) o-1->@:0/0 lens 368/264 e 0 to 0 dl 1308021481 ref 1 fl Interpret:/ffffffff/ffffffff rc -16/-1 LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 3 previous similar messages LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 35s LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 3 previous similar messages LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-16) req@cb011c2c x1371586166784254/t0(0) o-1->@:0/0 lens 368/264 e 0 to 0 dl 1308021516 ref 1 fl Interpret:/ffffffff/ffffffff rc -16/-1 LustreError: 11405:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 6 previous similar messages Lustre: 11405:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib recovering/t0 exp 00000000 cur 1308021506 last 0 Lustre: 11405:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 12 previous similar messages LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (a7954b84-49c0-eb44-edea-a997bb70078b): 1 clients in recovery for 0s LustreError: 11405:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 6 previous similar messages Lustre: 11404:0:(ldlm_lib.c:1559:target_recovery_overseer()) recovery is timed out, evict stale exports LustreError: 11404:0:(genops.c:1267:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d94eb03c-f960-e5f0-c46d-4cf0c55bff23@ Lustre: 11405:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 8589934655, current: 8589934657, replaying Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: 11404:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 11404:0:(mds_lov.c:1023:mds_notify()) Skipped 2 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0002: export for group 0 is changed: 0xe01aba00 -> 0xe3074e00 Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) Skipped 5 previous similar messages Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) changing the import d807a800 - df3d1000 Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: 8025:0:(filter.c:2550:filter_llog_connect()) lustre-OST0002: Recovery from log 0x27e484/0x0:1616adbe Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: 11405:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 11405:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 11 previous similar messages Lustre: 8026:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab80006802f012); group 0 Lustre: 8026:0:(filter.c:2846:filter_connect()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: == replay-single test 1: simple create =============================================================== 03:18:39 (1308046719) LustreError: 11741:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 5 previous similar messages Lustre: 11846:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 11846:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Release to readonly device sdb (0x800010): [inode 2614408] [block 1380480] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614409] [block 1380483] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614410] [block 1380486] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 1 previous similar message Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 12021:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 12884901891 LustreError: 12025:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 12025 Lustre: 12021:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 12021:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 12021:0:(mds_lov.c:1003:mds_notify()) Skipped 2 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8026:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab80006802f177); group 0 Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0001_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 2 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 4 previous similar messages Lustre: 12083:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 12083:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 1 previous similar message LustreError: 12029:0:(mdt_handler.c:2815:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.15@o2ib Lustre: 12029:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 12884901892, current: 12884901892, replaying LustreError: 12025:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 12884901889, ql: 1, comp: 1, conn: 2, next: 12884901892, last_committed: 12884901891) Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8026:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xf32d7600 -> 0xc3955c00 Lustre: 8026:0:(lustre_log.h:471:llog_group_set_export()) Skipped 5 previous similar messages Lustre: 8026:0:(llog_net.c:168:llog_receptor_accept()) changing the import e83d5000 - e0302800 Lustre: 8026:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) lustre-OST0002: Recovery from log 0x27e484/0x0:1616adbe Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) Skipped 3 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: Skipped 2 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 2a: touch ====================================================================== 03:18:56 (1308046736) LustreError: 12352:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 5 previous similar messages Lustre: 12457:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 12457:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Release to readonly device sdb (0x800010): [inode 2614409] [block 1380480] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614408] [block 1380992] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614410] [block 1381504] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 1 previous similar message Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 12636:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 17179869185 LustreError: 12640:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 12640 Lustre: 12636:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 12636:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 12636:0:(mds_lov.c:1003:mds_notify()) Skipped 2 previous similar messages Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0001_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 5 previous similar messages Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 2 previous similar messages LustreError: 12642:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 12642:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 2 previous similar messages Lustre: 12642:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 17179869186, current: 17179869188, replaying Lustre: 12642:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 3 previous similar messages LustreError: 12640:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xc3955c00 -> 0xf2d82800 Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) Skipped 5 previous similar messages Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) changing the import e0302800 - d19fdc00 Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: 8025:0:(filter.c:2550:filter_llog_connect()) lustre-OST0002: Recovery from log 0x27e484/0x0:1616adbe Lustre: 8025:0:(filter.c:2550:filter_llog_connect()) Skipped 1 previous similar message Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: Skipped 1 previous similar message Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 2b: touch ====================================================================== 03:19:12 (1308046752) LustreError: 12968:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 5 previous similar messages Lustre: 13073:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 13073:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message Release to readonly device sdb (0x800010): [inode 2614409] [block 1380483] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614410] [block 1380992] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 1 previous similar message Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 13247:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 21474836482 LustreError: 13251:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 13251 Lustre: 13247:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 13247:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 13247:0:(mds_lov.c:1003:mds_notify()) Skipped 2 previous similar messages Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8029:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab80006802fb33); group 0 Lustre: 8029:0:(filter.c:2846:filter_connect()) Skipped 5 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0001_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 5 previous similar messages Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 2 previous similar messages LustreError: 13253:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 13252:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@f3748800 x1371586212921597/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021597 ref 1 fl Interpret:/ffffffff/ffffffff rc -107/-1 LustreError: 13252:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 7 previous similar messages Lustre: 13252:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 21474836483, current: 21474836484, replaying Lustre: 13252:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 3 previous similar messages LustreError: 13253:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 1 previous similar message LustreError: 13251:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0002: export for group 0 is changed: 0xf75ae400 -> 0xf32d7e00 Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 5 previous similar messages Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) changing the import f4013c00 - d5ce3000 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) Skipped 2 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ============================================ 03:19:29 (1308046769) LustreError: 13579:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 5 previous similar messages Release to readonly device sdb (0x800010): [inode 2614411] [block 1380995] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614410] [block 1381504] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: MGS has stopped. Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: lustre-MDT0000: used disk, loading Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8025:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 1 previous similar message LustreError: 13870:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 13870:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 1 previous similar message Lustre: 13870:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 13870:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 41 previous similar messages Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xf32d7400 -> 0xdc4d6000 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) changing the import f7594c00 - ca742400 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: 8026:0:(filter.c:2550:filter_llog_connect()) lustre-OST0002: Recovery from log 0x27e484/0x0:1616adbe Lustre: 8026:0:(filter.c:2550:filter_llog_connect()) Skipped 2 previous similar messages Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) Skipped 9 previous similar messages Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ================================================= 03:19:43 (1308046783) LustreError: 13856:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 13870:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=114 *** Release to readonly device sdb (0x800010): [inode 2614410] [block 1381504] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614411] [block 1382016] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614412] [block 1382019] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 1 previous similar message LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 1 previous similar message LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 3 previous similar messages Lustre: Enabling ACL Lustre: Skipped 1 previous similar message Lustre: Enabling user_xattr Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: used disk, loading Lustre: 14547:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 30064771073 Lustre: 14547:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 1 previous similar message LustreError: 14552:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 14552 LustreError: 14552:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 1 previous similar message Lustre: 14547:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 14547:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 1 previous similar message Lustre: 14547:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 14547:0:(mds_lov.c:1003:mds_notify()) Skipped 5 previous similar messages Lustre: 8027:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8027:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 2 previous similar messages LustreError: 14545:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 14555:0:(mdt_handler.c:2815:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.18@o2ib Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 1 previous similar message Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xdc4d6000 -> 0xf3691400 Lustre: 8028:0:(llog_net.c:168:llog_receptor_accept()) changing the import ca742400 - dbf84800 Lustre: 8028:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) Skipped 3 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: Skipped 8 previous similar messages Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ================================================= 03:20:00 (1308046800) LustreError: 14881:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 14881:0:(osd_handler.c:935:osd_ro()) Skipped 1 previous similar message Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 14555:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=128 *** Lustre: 15056:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 15056:0:(quota_master.c:793:close_quota_files()) Skipped 5 previous similar messages Release to readonly device sdb (0x800010): [inode 2614410] [block 1382016] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614411] [block 1381504] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614412] [block 1381507] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 1 previous similar message Lustre: MGS has stopped. Lustre: Skipped 1 previous similar message Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 15169:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@f371a000 x1371559775175333/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 15169:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: e17ce6c0 (111542254400876/0/0/0) (rc: 0) Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 17 previous similar messages Lustre: 15292:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 15292:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 9 previous similar messages LustreError: 15227:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS Lustre: 15236:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 34359738369, current: 34359738369, replaying Lustre: 15236:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 15 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ====================================================== 03:20:14 (1308046814) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 17 previous similar messages Release to readonly device sdb (0x800010): [inode 2614410] [block 1381504] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614411] [block 1382016] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614412] [block 1382019] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 15782:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@dbfaa800 x1371559775175419/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LustreError: 15782:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: e17ce6c0 (111542254400876/0/0/0) (rc: 0) LustreError: 15851:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 15851:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 4 previous similar messages LustreError: 15849:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) lustre-OST0000: Recovery from log 0x27e482/0x0:1616adbc Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) Skipped 6 previous similar messages Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ============================================================ 03:20:27 (1308046827) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 15839:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 2 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 2 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 5 previous similar messages Lustre: 16452:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from d4b56168-b6b7-9abd-95b7-8869d7bbf4c8@0@lo t0 exp 00000000 cur 1308021637 last 0 Lustre: 16452:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 55 previous similar messages Lustre: Enabling ACL Lustre: Skipped 2 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 2 previous similar messages Lustre: 16460:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 42949672960 Lustre: 16460:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 2 previous similar messages LustreError: 16466:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 16466 LustreError: 16466:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 2 previous similar messages Lustre: 16460:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 16460:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 2 previous similar messages Lustre: 16460:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 16460:0:(mds_lov.c:1003:mds_notify()) Skipped 8 previous similar messages Lustre: 8028:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8028:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 7 previous similar messages Lustre: 8024:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab80006803179c); group 0 Lustre: 8024:0:(filter.c:2846:filter_connect()) Skipped 14 previous similar messages LustreError: 16466:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 38654705689, ql: 2, comp: 0, conn: 2, next: 42949672961, last_committed: 42949672960) Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 2 previous similar messages Lustre: 8024:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xf3702a00 -> 0xf2d60000 Lustre: 8028:0:(llog_net.c:168:llog_receptor_accept()) changing the import e618e400 - f3091c00 Lustre: 8028:0:(llog_net.c:168:llog_receptor_accept()) Skipped 17 previous similar messages Lustre: 8024:0:(lustre_log.h:471:llog_group_set_export()) Skipped 17 previous similar messages Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) ======================================================= 03:20:41 (1308046841) LustreError: 16795:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 16795:0:(osd_handler.c:935:osd_ro()) Skipped 2 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Alloc from readonly device sdb (0x800010): [inode 3660161] [logic 1] [goal 1903984] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Alloc from readonly device sdb (0x800010): [inode 3660161] [logic 2] [goal 1903986] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Alloc from readonly device sdb (0x800010): [inode 13] [logic 2] [goal 2167943] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] LustreError: 16452:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS Release to readonly device sdb (0x800010): [inode 2614414] [block 1382528] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614414] [block 1382017] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614413] [block 1382016] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614413] [block 1382529] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614415] [block 1382531] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 2 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 2 previous similar messages Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 17086:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 17086:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create =================================================== 03:21:02 (1308046862) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 17710:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 17710:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 5 previous similar messages LustreError: 17701:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 17701:0:(mgs_handler.c:678:mgs_handle()) Skipped 1 previous similar message Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) lustre-OST0002: Recovery from log 0x27e484/0x0:1616adbe Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) Skipped 7 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: Skipped 15 previous similar messages Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ================================================================== 03:21:17 (1308046877) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 3660174] [block 1903989] [count 1] [is_meta 1] Lustre: 18148:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 18148:0:(quota_master.c:793:close_quota_files()) Skipped 9 previous similar messages Release to readonly device sdb (0x800010): [inode 2614418] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614417] [block 1383043] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Lustre: Failing over lustre-MDT0000-mdtlov Lustre: Skipped 28 previous similar messages Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775175979 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308021677] [real_sent 1308021677] [current 1308021689] [deadline 12s] [delay 0s] req@e83d8000 x1371559775175979/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308021689 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@dc822c00 x1371559775175983/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@e2ef7400 x1371559775175989/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775175982 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308021689] [real_sent 1308021689] [current 1308021700] [deadline 11s] [delay 0s] req@c5ecb800 x1371559775175982/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308021700 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 18261:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 29 previous similar messages LustreError: 18336:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@cba53800 x1371586166786432/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021731 ref 1 fl Interpret:/ffffffff/ffffffff rc -107/-1 Lustre: 18335:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 55834574849, current: 55834574850, replaying Lustre: 18335:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 501 previous similar messages LustreError: 18336:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 24 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 14 previous similar messages Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ================================================== 03:21:44 (1308046904) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1383555] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 3 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 3 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 3 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 7 previous similar messages Lustre: 18942:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->192.168.4.128@o2ib netid 90000: select flavor null Lustre: 18942:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 74 previous similar messages Lustre: Enabling ACL Lustre: Skipped 3 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 3 previous similar messages Lustre: 18945:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 60129542145 Lustre: 18945:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 3 previous similar messages LustreError: 18951:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 18951 LustreError: 18951:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 3 previous similar messages Lustre: 18945:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 18945:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 3 previous similar messages Lustre: 18945:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 18945:0:(mds_lov.c:1003:mds_notify()) Skipped 11 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 9 previous similar messages LustreError: 18942:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 18942:0:(mgs_handler.c:678:mgs_handle()) Skipped 1 previous similar message Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 3 previous similar messages Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xdf0f2e00 -> 0xc9471400 Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) changing the import d7448800 - f36ad000 Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) Skipped 23 previous similar messages Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 23 previous similar messages Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ======================================================== 03:22:01 (1308046921) LustreError: 19285:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 19285:0:(osd_handler.c:935:osd_ro()) Skipped 3 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1383555] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 3 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 3 previous similar messages Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 19504:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@f38c5800 x1371559775176185/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 13 previous similar messages LustreError: 19504:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d76ef4c0 (111542254400876/0/0/0) (rc: 0) Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ================================================== 03:22:14 (1308046934) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1383043] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 20123:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@c6c30c00 x1371559775176277/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LustreError: 20123:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d76ef2c0 (111542254400876/0/0/0) (rc: 0) Lustre: 20246:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 20246:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 15 previous similar messages LustreError: 20192:0:(mdt_handler.c:2815:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 20192:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 6 previous similar messages Lustre: 8026:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8026:0:(filter.c:2550:filter_llog_connect()) Skipped 12 previous similar messages Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink =================================================== 03:22:31 (1308046951) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1384064] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 20741:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@e3037000 x1371559775176368/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LustreError: 20741:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d76ef2c0 (111542254400876/0/0/0) (rc: 0) Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================== 03:22:48 (1308046968) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614416] [block 1383040] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1383043] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 21360:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 21360:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d76ef2c0 (111542254400876/0/0/0) (rc: 0) Lustre: 8024:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab80006803c0d0); group 0 Lustre: 8024:0:(filter.c:2846:filter_connect()) Skipped 23 previous similar messages LustreError: 21425:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 21425:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ===================================================== 03:23:01 (1308046981) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1383043] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 21980:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 21980:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: e6154cc0 (111542254400876/0/0/0) (rc: 0) LustreError: 22037:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 22037:0:(mgs_handler.c:678:mgs_handle()) Skipped 5 previous similar messages LustreError: 22045:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 81604378630, ql: 2, comp: 0, conn: 2, next: 81604378632, last_committed: 81604378631) Lustre: 22045:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: 22045:0:(mdd_lov.c:635:mdd_lov_destroy()) Get lov ea failed for [0x200000bd0:0x109:0x0] rc = 0 Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close =============================================== 03:23:18 (1308046998) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614419] [block 1384576] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1384064] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614417] [block 1383552] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 22669:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 85899345923, ql: 2, comp: 0, conn: 2, next: 85899345924, last_committed: 85899345923) Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: Skipped 22 previous similar messages Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ============================================ 03:23:35 (1308047015) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 42 previous similar messages Lustre: 23109:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 23109:0:(quota_master.c:793:close_quota_files()) Skipped 15 previous similar messages Release to readonly device sdb (0x800010): [inode 2614418] [block 1384064] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@f72f7c00 x1371559775176757/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 23222:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 23222:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: e6154ec0 (111542254400876/0/0/0) (rc: 1) LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 17 previous similar messages LustreError: 23290:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 90194313219, ql: 2, comp: 0, conn: 2, next: 90194313221, last_committed: 90194313220) Lustre: 23290:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 23290:0:(mds_lov.c:1023:mds_notify()) Skipped 50 previous similar messages Lustre: 23290:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ================================ 03:23:51 (1308047031) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614419] [block 1384576] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614418] [block 1384064] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 23916:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 94489280513, current: 94489280517, replaying Lustre: 23916:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 56 previous similar messages LustreError: 23914:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 94489280513, ql: 2, comp: 0, conn: 2, next: 94489280515, last_committed: 94489280514) LustreError: 23914:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: 23914:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 24 previous similar messages Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================== 03:24:08 (1308047048) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614420] [block 1384579] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614419] [block 1384576] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 8 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 8 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 17 previous similar messages Lustre: Enabling ACL Lustre: Skipped 8 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 8 previous similar messages Lustre: 24532:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 98784247810 Lustre: 24532:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 8 previous similar messages LustreError: 24539:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 24539 LustreError: 24539:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 8 previous similar messages Lustre: 24532:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 24532:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 8 previous similar messages Lustre: 24532:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 24532:0:(mds_lov.c:1003:mds_notify()) Skipped 26 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 25 previous similar messages Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 8 previous similar messages Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xc6cb8400 -> 0xcb26d200 Lustre: 8027:0:(llog_net.c:168:llog_receptor_accept()) changing the import cdd0e000 - dbaeb400 Lustre: 8027:0:(llog_net.c:168:llog_receptor_accept()) Skipped 53 previous similar messages Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 53 previous similar messages Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================== 03:24:22 (1308047062) LustreError: 24875:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 24875:0:(osd_handler.c:935:osd_ro()) Skipped 8 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614420] [block 1385088] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614419] [block 1384576] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614421] [block 1384579] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 8 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 8 previous similar messages Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 25165:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 25165:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 16 previous similar messages LustreError: 25163:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: DEBUG MARKER: == replay-single test 18: |X| open(O_CREAT), unlink, touch new, close, touch, unlink ================= 03:24:39 (1308047079) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614420] [block 1385088] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614421] [block 1384579] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ca604800 x1371559775177187/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 25720:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 25720:0:(ldlm_resource.c:748:ldlm_resource_complain()) Skipped 3 previous similar messages LustreError: 25720:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: cd318ec0 (111542254400876/0/0/0) (rc: 1) LustreError: 25720:0:(ldlm_resource.c:754:ldlm_resource_complain()) Skipped 3 previous similar messages LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: 25792:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib recovering/t107374182403 exp f3083e00 cur 1308021893 last 1308021891 Lustre: 25792:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 108 previous similar messages Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) Skipped 26 previous similar messages Lustre: DEBUG MARKER: == replay-single test 19: |X| mcreate, open, write, rename =========================================== 03:24:55 (1308047095) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614422] [block 1385600] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 137-5: UUID 'lustre-MDT0000_UUID' is not available for connect (not set up) LustreError: 26413:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 26413:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ======= 03:25:13 (1308047113) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614424] [block 1385603] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614423] [block 1386112] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 27039:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 111669149713, ql: 2, comp: 0, conn: 2, next: 115964116993, last_committed: 115964116992) Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay, (test mds_cleanup_orphans) =============== 03:25:26 (1308047126) Lustre: 27311:0:(genops.c:1378:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7954b84-49c0-eb44-edea-a997bb70078b at adminstrative request LustreError: 8026:0:(ldlm_resource.c:1084:ldlm_resource_get()) lvbo_init failed for resource 930: rc -2 LustreError: 16944:0:(filter_io.c:723:filter_preprw_write()) lustre-OST0001: BRW to missing obj 930/0:rc -2 LustreError: 16944:0:(filter_io.c:723:filter_preprw_write()) lustre-OST0001: BRW to missing obj 930/0:rc -2 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: before 1302972, after 1302976 Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 03:25:46 (1308047146) Lustre: 27993:0:(genops.c:1378:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7954b84-49c0-eb44-edea-a997bb70078b at adminstrative request Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 03:25:49 (1308047149) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614426] [block 1386624] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614425] [block 1387138] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614424] [block 1387136] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614424] [block 1387141] [count 1] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 28484:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 28493:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@f40cd000 x1371586212922239/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308021992 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 28493:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 40 previous similar messages Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ======== 03:26:05 (1308047165) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614426] [block 1386630] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614425] [block 1386627] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 29110:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->192.168.4.128@o2ib netid 90000: select flavor null Lustre: 29110:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 155 previous similar messages Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 03:26:22 (1308047182) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614427] [block 1388160] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614426] [block 1387648] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ============ 03:26:39 (1308047199) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614427] [block 1387651] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614428] [block 1387654] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 30440:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 30440:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 31 previous similar messages Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ============ 03:26:55 (1308047215) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614429] [block 1389184] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614428] [block 1388672] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 31013:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 141733920773, ql: 2, comp: 0, conn: 2, next: 141733920774, last_committed: 141733920773) Lustre: 31013:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 03:27:09 (1308047229) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614430] [block 1388678] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 31579:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@f3a38800 x1371559775178282/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 12018:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: 31579:0:(ldlm_resource.c:748:ldlm_resource_complain()) Skipped 3 previous similar messages LustreError: 31579:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: cb911280 (111542254400876/0/0/0) (rc: 0) LustreError: 31579:0:(ldlm_resource.c:754:ldlm_resource_complain()) Skipped 3 previous similar messages Lustre: 8028:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000680415fd); group 0 Lustre: 8028:0:(filter.c:2846:filter_connect()) Skipped 46 previous similar messages LustreError: 31646:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 141733920772, ql: 2, comp: 0, conn: 2, next: 146028888066, last_committed: 146028888065) LustreError: 31646:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 31646:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 03:27:26 (1308047246) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614431] [block 1389696] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 03:27:42 (1308047262) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614433] [block 1390208] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 471:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 154618822662, ql: 2, comp: 0, conn: 2, next: 154618822663, last_committed: 154618822662) Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 48 previous similar messages Lustre: Skipped 0 previous similar message Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 03:27:59 (1308047279) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 95 previous similar messages Lustre: 989:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 989:0:(quota_master.c:793:close_quota_files()) Skipped 31 previous similar messages Release to readonly device sdb (0x800010): [inode 2614435] [block 1390211] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 92 previous similar messages Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 03:28:16 (1308047296) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614437] [block 1391744] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 1817:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 163208757257, current: 163208757259, replaying Lustre: 1817:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 123 previous similar messages LustreError: 1815:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 163208757255, ql: 2, comp: 0, conn: 2, next: 163208757257, last_committed: 163208757256) Lustre: 1815:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: lustre-OST0002: received MDS connection from 0@lo Lustre: Skipped 47 previous similar messages Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ====================================================================================================== 03:28:33 (1308047313) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614439] [block 1392256] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 15 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 15 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 15 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 31 previous similar messages Lustre: Enabling ACL Lustre: Skipped 15 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 15 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 15 previous similar messages Lustre: 2459:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 167503724551 Lustre: 2459:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 15 previous similar messages LustreError: 2465:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 2465 LustreError: 2465:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 15 previous similar messages Lustre: 2459:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 2459:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 15 previous similar messages Lustre: 2459:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 2459:0:(mds_lov.c:1003:mds_notify()) Skipped 47 previous similar messages Lustre: 8027:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8027:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 47 previous similar messages LustreError: 2465:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 167503724551, ql: 2, comp: 0, conn: 2, next: 167503724552, last_committed: 167503724551) Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 15 previous similar messages Lustre: 2465:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: 8027:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0002: export for group 0 is changed: 0xf6b13400 -> 0xf4ffdc00 Lustre: 8027:0:(lustre_log.h:471:llog_group_set_export()) Skipped 91 previous similar messages Lustre: 8027:0:(llog_net.c:168:llog_receptor_accept()) changing the import e836a800 - e87b7000 Lustre: 8027:0:(llog_net.c:168:llog_receptor_accept()) Skipped 95 previous similar messages Lustre: 2465:0:(mdd_orphans.c:359:orph_key_test_and_del()) Skipped 1 previous similar message Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ============= 03:28:47 (1308047327) Lustre: 2753:0:(genops.c:1378:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a7954b84-49c0-eb44-edea-a997bb70078b at adminstrative request Lustre: DEBUG MARKER: SKIP: replay-single test_33a skipping ALWAYS excluded test 33a Lustre: DEBUG MARKER: SKIP: replay-single test_33b skipping ALWAYS excluded test 33b Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ======== 03:28:50 (1308047330) LustreError: 3037:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 3037:0:(osd_handler.c:935:osd_ro()) Skipped 14 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614440] [block 1392259] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 15 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 15 previous similar messages Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 3306:0:(lov_ea.c:215:lsm_unpackmd_v1()) OST index 2 more than OST count 0 Lustre: 3306:0:(lov_pack.c:64:lov_dump_lmm_common()) objid 0x1, magic 0x0bd10bd0, pattern 0x1 Lustre: 3306:0:(lov_pack.c:67:lov_dump_lmm_common()) stripe_size 1048576, stripe_count 1 Lustre: 3306:0:(lov_pack.c:84:lov_dump_lmm_objects()) stripe 0 idx 2 subobj 0x0/0x522 LustreError: 3245:0:(mdt_handler.c:5518:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000 LustreError: 3311:0:(mdt_handler.c:2815:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 3311:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 33 previous similar messages Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ====================================== 03:28:57 (1308047337) LustreError: 3455:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=119 *** LustreError: 3455:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@cd53f400 x1371586166787615/t176093659141(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 456/576 e 0 to 0 dl 1308022142 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 3782:0:(mdt_handler.c:5518:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ======================================================== 03:29:07 (1308047347) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614444] [block 1394816] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614443] [block 1394304] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614442] [block 1393280] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 7620:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) ldlm_cancel from 192.168.4.15@o2ib arrived at 1308022157 with bad export cookie 8118723492498915129 Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ====================================================================================================== 03:29:20 (1308047360) Lustre: 8025:0:(filter.c:2550:filter_llog_connect()) lustre-OST0000: Recovery from log 0x27e482/0x0:1616adbc Lustre: 8025:0:(filter.c:2550:filter_llog_connect()) Skipped 50 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614444] [block 1393286] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614443] [block 1393283] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 5239:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: 5239:0:(mdd_orphans.c:359:orph_key_test_and_del()) Skipped 1 previous similar message LustreError: 5153:0:(mdt_handler.c:5518:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================= 03:29:26 (1308047366) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614443] [block 1393283] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================= 03:29:47 (1308047387) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Alloc from readonly device sdb (0x800010): [inode 2614443] [logic 4] [goal 1395878] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] Alloc from readonly device sdb (0x800010): [inode 2614448] [logic 4] [goal 1395879] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] Alloc from readonly device sdb (0x800010): [inode 2614447] [logic 4] [goal 1395880] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] Removing read-only on unknown block (0x800010) LustreError: 7618:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) llog_origin_handle_cancel from 0@lo arrived at 1308022194 with bad export cookie 8118723492498962834 LustreError: 7618:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) Skipped 2 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 03:30:09 (1308047409) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 7234:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=117 *** LustreError: 7225:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 7225:0:(mgs_handler.c:678:mgs_handle()) Skipped 9 previous similar messages Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 03:30:43 (1308047443) LustreError: 7618:0:(filter.c:3135:__filter_oa2dentry()) lustre-OST0001: filter_sync on non-existent object: 930:0 LustreError: 7618:0:(ost_handler.c:1802:ost_blocking_ast()) Error -2 syncing data on lock cancel Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: lustre-OST0001-osc-MDT0000: Connection to service lustre-OST0001 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: 8026:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-OST0001: lustre-MDT0000-mdtlov_UUID reconnecting LustreError: 167-0: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: lustre-OST0001-osc-MDT0000: Connection restored to service lustre-OST0001 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ================================================= 03:30:44 (1308047444) LustreError: 8112:0:(filter.c:4564:filter_iocontrol()) *** setting device unknown-block(8,33) read-only *** Turning device sdc (0x800021) read-only Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Removing read-only on unknown block (0x800021) LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LDISKFS-fs (sdc1): recovery complete LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 8415:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 8358:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================= 03:31:44 (1308047504) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775183157 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308022305] [real_sent 1308022305] [current 1308022312] [deadline 7s] [delay 0s] req@c7e17c00 x1371559775183157/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308022312 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 9043:0:(import.c:323:ptlrpc_invalidate_import()) MGS: rc = -110 waiting for callback (1 != 0) LustreError: 9043:0:(import.c:349:ptlrpc_invalidate_import()) @@@ still on sending list req@c8988800 x1371559775183159/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 0 dl 1308022318 ref 1 fl Rpc:N/ffffffff/ffffffff rc 0/-1 LustreError: 9043:0:(import.c:365:ptlrpc_invalidate_import()) MGS: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out. Lustre: 9043:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s LustreError: 8026:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=204 *** Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ============================================= 03:32:12 (1308047532) LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9101:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping Lustre: 9209:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775183185 sent from lustre-OST0000-osc-MDT0000 to NID 0@lo has timed out for slow reply: [sent 1308022320] [real_sent 1308022320] [current 1308022346] [deadline 26s] [delay 0s] req@db590800 x1371559775183185/t0(0) o-1->lustre-OST0000_UUID@192.168.4.128@o2ib:28/4 lens 400/400 e 0 to 1 dl 1308022346 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 9209:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 9209:0:(osc_create.c:605:osc_create()) lustre-OST0000-osc-MDT0000: oscc recovery failed: -11 LustreError: 8024:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 8024:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-OST0000: lustre-MDT0000-mdtlov_UUID reconnecting LustreError: 9209:0:(lov_obd.c:1068:lov_clear_orphans()) error in orphan recovery on OST idx 0/3: rc = -11 Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. LustreError: 9209:0:(mds_lov.c:879:__mds_lov_synchronize()) lustre-OST0000_UUID failed at mds_lov_clear_orphans: -11 LustreError: 9209:0:(mds_lov.c:900:__mds_lov_synchronize()) lustre-OST0000_UUID sync failed -11, deactivating LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 1 previous similar message Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 LustreError: 9115:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 1 previous similar message LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800004 took longer than estimated (20:4s); client may timeout. req@db59b000 x1371586166800004/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022392 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 2 previous similar messages Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib t201863465330 exp d30e2a00 cur 1308022431 last 1308022431 Lustre: 9115:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 164 previous similar messages Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800032 took longer than estimated (20:9s); client may timeout. req@d6482000 x1371586166800032/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022422 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) Skipped 1 previous similar message LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 2 previous similar messages Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800064 took longer than estimated (20:9s); client may timeout. req@d2714c2c x1371586166800064/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022457 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) Skipped 1 previous similar message LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 2 previous similar messages Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 LustreError: 9669:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-16) req@d271442c x1371586166800122/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022521 ref 1 fl Interpret:/ffffffff/ffffffff rc -16/-1 LustreError: 9669:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 58 previous similar messages Lustre: 9115:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 9115:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 192 previous similar messages Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800099 took longer than estimated (20:9s); client may timeout. req@e816b000 x1371586166800099/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022492 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800134 took longer than estimated (20:9s); client may timeout. req@e33bd400 x1371586166800134/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022527 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(libcfs_fail.h:135:cfs_race()) cfs_race id 701 sleeping LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) cfs_fail_race id 701 waking LustreError: 9669:0:(libcfs_fail.h:139:cfs_race()) Skipped 3 previous similar messages LustreError: 9115:0:(libcfs_fail.h:137:cfs_race()) cfs_fail_race id 701 awake Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9669:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 5 previous similar messages Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800169 took longer than estimated (20:9s); client may timeout. req@d56ab400 x1371586166800169/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022562 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 Lustre: 9115:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:846:target_handle_connect()) lustre-MDT0000: refuse reconnection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib to 0xd30e2a00/1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800204 took longer than estimated (20:9s); client may timeout. req@d271682c x1371586166800204/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022597 ref 1 fl Complete:/ffffffff/ffffffff rc -114/-1 Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ============================================= 03:36:54 (1308047814) LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800239 took longer than estimated (20:20s); client may timeout. req@df60b800 x1371586166800239/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022632 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800284 took longer than estimated (20:20s); client may timeout. req@d2717c2c x1371586166800284/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022677 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 7 previous similar messages LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) Skipped 1 previous similar message LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) Skipped 2 previous similar messages LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800371 took longer than estimated (20:20s); client may timeout. req@db593c00 x1371586166800371/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022767 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) Skipped 1 previous similar message LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) Skipped 2 previous similar messages LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) Skipped 4 previous similar messages LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) @@@ Request x1371586166800502 took longer than estimated (20:20s); client may timeout. req@c8699800 x1371586166800502/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308022902 ref 1 fl Complete:/ffffffff/ffffffff rc 0/-1 Lustre: 9115:0:(service.c:1728:ptlrpc_server_handle_request()) Skipped 2 previous similar messages LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 9115:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 11 previous similar messages LustreError: 9115:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms LustreError: 9115:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) lustre-MDT0000: exp d30e2a00 already connecting Lustre: 9669:0:(ldlm_lib.c:785:target_handle_connect()) Skipped 12 previous similar messages Lustre: 9115:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from a7954b84-49c0-eb44-edea-a997bb70078b@192.168.4.15@o2ib t201863465330 exp d30e2a00 cur 1308023057 last 1308023056 Lustre: 9115:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 28 previous similar messages Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ============================================= 03:44:24 (1308048264) LustreError: 11451:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 11451:0:(osd_handler.c:935:osd_ro()) Skipped 5 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 66 previous similar messages Lustre: 11593:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 11593:0:(quota_master.c:793:close_quota_files()) Skipped 21 previous similar messages Release to readonly device sdb (0x800010): [inode 2614456] [block 1400482] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614456] [block 1400485] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614457] [block 1399968] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614455] [block 1400480] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614455] [block 1400484] [count 1] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 7 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 7 previous similar messages Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 9 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 8 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 17 previous similar messages Lustre: Enabling ACL Lustre: Skipped 8 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 8 previous similar messages Lustre: 11733:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 11733:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 8 previous similar messages Lustre: 11733:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 11733:0:(mds_lov.c:1003:mds_notify()) Skipped 26 previous similar messages Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8029:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 26 previous similar messages Lustre: 8029:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab800068079c94); group 0 Lustre: 8029:0:(filter.c:2846:filter_connect()) Skipped 41 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 32 previous similar messages Lustre: 8026:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xf7172a00 -> 0xcc225c00 Lustre: 8026:0:(lustre_log.h:471:llog_group_set_export()) Skipped 53 previous similar messages Lustre: 8026:0:(llog_net.c:168:llog_receptor_accept()) changing the import c5fd7c00 - f685b000 Lustre: 8026:0:(llog_net.c:168:llog_receptor_accept()) Skipped 59 previous similar messages Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 35 previous similar messages Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) Skipped 19 previous similar messages LustreError: 11671:0:(mdt_handler.c:5518:mdt_iocontrol()) Aborting recovery for device lustre-MDT0000 Lustre: 11856:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 11856:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 34 previous similar messages LustreError: 11740:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 11739:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=712 *** LustreError: 11895:0:(service.c:855:ptlrpc_check_req()) @@@ Invalid replay without recovery req@d6482400 x1371586166800683/t0(206158430209) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 472/0 e 0 to 0 dl 0 ref 2 fl New:/ffffffff/ffffffff rc 0/-1 LustreError: 11740:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 14 previous similar messages LustreError: 11728:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 12148:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 210453397504 Lustre: 12148:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 6 previous similar messages LustreError: 12152:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 12152 LustreError: 12152:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 6 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 47 previous similar messages Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 6 previous similar messages Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ======================================================== 03:44:45 (1308048285) Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ============================ 03:44:46 (1308048286) LustreError: 12164:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=122 *** LustreError: 12164:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@d2157400 x1371586166800827/t0(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 232/216 e 0 to 0 dl 1308023091 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 Lustre: 12156:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 12156:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 50 previous similar messages Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775184484 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308023116] [real_sent 1308023116] [current 1308023123] [deadline 7s] [delay 0s] req@e834d000 x1371559775184484/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308023123 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 12846:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@e7c8442c x1371586166800872/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308023133 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 12846:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 42 previous similar messages Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775184486 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308023123] [real_sent 1308023123] [current 1308023129] [deadline 6s] [delay 0s] req@d37d7c00 x1371559775184486/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308023129 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 12788:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s Lustre: 12860:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 214748364803, current: 214748364805, replaying Lustre: 12860:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 564 previous similar messages LustreError: 12856:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 214748364803, ql: 1, comp: 1, conn: 2, next: 214748364805, last_committed: 214748364804) Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) =========================== 03:45:35 (1308048335) Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: Skipped 3 previous similar messages LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_connect operation failed with -19 LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 13366:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 13309:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) =========================== 03:46:53 (1308048413) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 14040:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 219043332197, ql: 2, comp: 0, conn: 2, next: 219043332200, last_committed: 219043332199) LustreError: 14040:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 14040:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 4 previous similar messages LustreError: 8025:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=216 *** LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_create operation failed with -30 LustreError: 14178:0:(osc_create.c:605:osc_create()) lustre-OST0000-osc-MDT0000: oscc recovery failed: -30 LustreError: 14178:0:(lov_obd.c:1068:lov_clear_orphans()) error in orphan recovery on OST idx 0/3: rc = -30 LustreError: 14178:0:(mds_lov.c:879:__mds_lov_synchronize()) lustre-OST0000_UUID failed at mds_lov_clear_orphans: -30 LustreError: 14178:0:(mds_lov.c:900:__mds_lov_synchronize()) lustre-OST0000_UUID sync failed -30, deactivating Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ================================== 03:47:10 (1308048430) Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ================================================ 03:47:17 (1308048437) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ======================== 03:47:34 (1308048454) LustreError: 15212:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=115 *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614463] [block 1404066] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614462] [block 1404064] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614462] [block 1403555] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614461] [block 1403552] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================= 03:47:52 (1308048472) LustreError: 15983:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=107 *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614461] [block 1403552] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614461] [block 1404065] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614462] [block 1404064] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614462] [block 1403553] [count 2] [is_meta 1] LustreError: 15973:0:(ldlm_lockd.c:1141:ldlm_handle_enqueue0()) ### lock on disconnected export c7e09a00 ns: MGS lock: f5c73cc0/0x70ab80006807d233 lrc: 2/0,0 mode: --/CR res: 111542254400876/0 rrc: 5 type: PLN flags: 0x0 remote: 0x70ab80006807d22c expref: -99 pid: 15973 timeout 0 LustreError: 15973:0:(mgs_handler.c:783:mgs_handle()) MGS handle cmd=101 rc=-107 LustreError: 11-0: an error occurred while communicating with 0@lo. The ldlm_enqueue operation failed with -107 LustreError: 14033:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@d3265000 x1371559775185743/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 14033:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Removing read-only on unknown block (0x800010) Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775185742 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308023273] [real_sent 1308023273] [current 1308023284] [deadline 11s] [delay 0s] req@ddfb0800 x1371559775185742/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308023284 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 16651:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 11s LustreError: 16716:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 231928233985, ql: 1, comp: 1, conn: 2, next: 231928233998, last_committed: 231928233997) Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ======= 03:48:11 (1308048491) LustreError: 16718:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=107 *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614464] [block 1405091] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614461] [block 1405088] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614461] [block 1403553] [count 2] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 137-5: UUID 'lustre-MDT0000_UUID' is not available for connect (not set up) LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: == replay-single test 53d: |X| close reply while two MDC requests in flight ========================== 03:48:31 (1308048511) LustreError: 17466:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=13b *** LustreError: 17466:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=13b *** LustreError: 17466:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@d9a01800 x1371586166801610/t240518168591(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 360/424 e 0 to 0 dl 1308023316 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight =========================== 03:48:49 (1308048529) LustreError: 18098:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=119 *** LustreError: 18098:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@df6d8c00 x1371586166801678/t244813135886(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 456/416 e 0 to 0 dl 1308023352 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614464] [block 1405600] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614466] [block 1406112] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 18766:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 14033:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@d6524c00 x1371559775186478/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 14033:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 18766:0:(ldlm_resource.c:748:ldlm_resource_complain()) Skipped 4 previous similar messages LustreError: 18766:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: da442a80 (111542254400876/0/0/0) (rc: 0) LustreError: 18766:0:(ldlm_resource.c:754:ldlm_resource_complain()) Skipped 4 previous similar messages LustreError: 18833:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 244813135873, ql: 1, comp: 1, conn: 2, next: 244813135887, last_committed: 244813135886) Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight =========== 03:49:04 (1308048544) LustreError: 18835:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=119 *** LustreError: 18835:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@e3040400 x1371586166801738/t249108103182(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 456/416 e 0 to 0 dl 1308023367 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 LustreError: 18837:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=13b *** Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614466] [block 1405600] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614466] [block 1407137] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614467] [block 1405603] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ====================================================================================================== 03:49:27 (1308048567) LustreError: 19575:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=119 *** LustreError: 19575:0:(libcfs_fail.h:81:cfs_fail_check_set()) Skipped 1 previous similar message LustreError: 19575:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@dc80f800 x1371586166801807/t253403070478(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 456/416 e 0 to 0 dl 1308023391 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 LustreError: 19575:0:(ldlm_lib.c:2113:target_send_reply_msg()) Skipped 1 previous similar message LustreError: 19577:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=115 *** LustreError: 19577:0:(libcfs_fail.h:81:cfs_fail_check_set()) Skipped 1 previous similar message Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614468] [block 1407650] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1407652] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614467] [block 1407648] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614467] [block 1406113] [count 2] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 53h: |X| open request and close reply while two MDC requests in flight ========= 03:49:48 (1308048588) LustreError: 20387:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=107 *** LustreError: 20389:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=13b *** LustreError: 20389:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=13b *** LustreError: 20389:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@cb69f800 x1371586166801874/t257698037774(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 360/424 e 0 to 0 dl 1308023393 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614468] [block 1408672] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1408164] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614467] [block 1408160] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614467] [block 1408673] [count 2] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 03:50:08 (1308048608) LustreError: 21129:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=12b *** LustreError: 21129:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=30c *** LustreError: 21129:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@c6cfd000 x1371586166801930/t261993005068(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 544/544 e 0 to 0 dl 1308023431 ref 1 fl Interpret:/ffffffff/ffffffff rc 301/-1 LustreError: 21129:0:(mdt_open.c:915:mdt_reconstruct_open()) This is reconstruct open: disp=0x17, result=0 Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ================================= 03:50:55 (1308048655) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614470] [block 1409187] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1409184] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ===================================== 03:51:19 (1308048679) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614470] [block 1410208] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614470] [block 1409699] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614471] [block 1409696] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1409184] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1410209] [count 2] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ================ 03:51:34 (1308048694) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614471] [block 1409187] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614468] [block 1409184] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ================================================ 03:52:16 (1308048736) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 23696:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) ldlm_cancel from 192.168.4.15@o2ib arrived at 1308023538 with bad export cookie 8118723492499233895 Removing read-only on unknown block (0x800010) LustreError: 23696:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) ldlm_cancel from 192.168.4.18@o2ib arrived at 1308023545 with bad export cookie 8118723492499233902 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ============================================ 03:52:42 (1308048762) LustreError: 24200:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=123 *** Lustre: 24525:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: a7954b84-49c0-eb44-edea-a997bb70078b reconnecting Lustre: 24525:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 10 previous similar messages LustreError: 24525:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=119 *** LustreError: 24525:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@f2d6dc00 x1371586166819035/t279172874248(0) o-1->a7954b84-49c0-eb44-edea-a997bb70078b@NET_0x50000c0a8040f_UUID:0/0 lens 368/408 e 0 to 0 dl 1308023622 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 03:53:55 (1308048835) Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: Skipped 1 previous similar message LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 25017:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 24960:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: Skipped 1 previous similar message LustreError: 25099:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 507 sleeping for 10000ms LustreError: 25099:0:(fail.c:126:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 25099:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 507 awake LustreError: 25099:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 23698:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) llog_origin_handle_cancel from 0@lo arrived at 1308023663 with bad export cookie 8118723492499440052 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 25365:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from d4b56168-b6b7-9abd-95b7-8869d7bbf4c8@0@lo t0 exp 00000000 cur 1308023664 last 0 Lustre: 25365:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 158 previous similar messages Lustre: 25369:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0001_UUID Lustre: 25369:0:(mds_lov.c:1003:mds_notify()) Skipped 51 previous similar messages Lustre: 8024:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0001: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8024:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 51 previous similar messages Lustre: 8024:0:(filter.c:2846:filter_connect()) lustre-OST0001: Received MDS connection (0x70ab8000680c8c22); group 0 Lustre: 8024:0:(filter.c:2846:filter_connect()) Skipped 57 previous similar messages Lustre: 25431:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 25431:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 37 previous similar messages LustreError: 25375:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 25375:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 31 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xf5677800 -> 0xd31a3000 Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 101 previous similar messages Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) changing the import c4736400 - cbb2d000 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) Skipped 103 previous similar messages Lustre: Skipped 56 previous similar messages Lustre: 8027:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: Skipped 51 previous similar messages Lustre: 8027:0:(filter.c:2550:filter_llog_connect()) Skipped 52 previous similar messages LustreError: 25364:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 25364:0:(mgs_handler.c:678:mgs_handle()) Skipped 25 previous similar messages Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ================================ 03:54:48 (1308048888) LustreError: 25810:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 25810:0:(osd_handler.c:935:osd_ro()) Skipped 13 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 109 previous similar messages Lustre: 25915:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 25915:0:(quota_master.c:793:close_quota_files()) Skipped 35 previous similar messages Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 17 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 17 previous similar messages Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 19 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 17 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 17 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 35 previous similar messages Lustre: Enabling ACL Lustre: Skipped 17 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 17 previous similar messages Lustre: 26089:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 283467841937 Lustre: 26089:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 18 previous similar messages LustreError: 26095:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 26095 LustreError: 26095:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 18 previous similar messages Lustre: 26089:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 26089:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 17 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 101 previous similar messages Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 03:55:04 (1308048904) LustreError: 26493:0:(filter.c:4564:filter_iocontrol()) *** setting device unknown-block(8,33) read-only *** Turning device sdc (0x800021) read-only Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) LustreError: Skipped 3 previous similar messages Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Removing read-only on unknown block (0x800021) Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_connect operation failed with -19 LustreError: Skipped 1 previous similar message LDISKFS-fs (sdc1): recovery complete LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 26804:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 26747:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 8024:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000->NET_0x50000c0a80412_UUID netid 50000: select flavor null Lustre: 8024:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 205 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. LustreError: 26888:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 221 sleeping for 30000ms LustreError: 8027:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@cad2002c x1371586212924375/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308023757 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 8027:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 83 previous similar messages LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) LustreError: Skipped 4 previous similar messages LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 16s Lustre: 7272:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: Skipped 14 previous similar messages LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 27131:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 27074:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 03:56:47 (1308049007) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 27771:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 13a sleeping for 60000ms LustreError: 27771:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 13a awake LustreError: 27771:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 11-0: an error occurred while communicating with 0@lo. The llog_origin_handle_next_block operation failed with -19 LustreError: Skipped 5 previous similar messages LustreError: 27901:0:(llog_cat.c:485:llog_cat_process_thread()) llog_cat_process() failed -19 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 03:58:37 (1308049117) LustreError: 8257:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 222 sleeping for 30000ms Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: Skipped 3 previous similar messages LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 6s Lustre: 28657:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 28600:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: 7272:0:(import.c:529:import_select_connection()) Skipped 3 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay =============================================== 03:59:14 (1308049154) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614482] [block 1416896] [count 1] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614482] [block 1416385] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614481] [block 1415875] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614479] [block 1416899] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775194191 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308023958] [real_sent 1308023958] [current 1308023965] [deadline 7s] [delay 0s] req@dbda2800 x1371559775194191/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308023965 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775194193 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308023965] [real_sent 1308023965] [current 1308023971] [deadline 6s] [delay 0s] req@e4bd1c00 x1371559775194193/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308023971 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: 29377:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 296352743432, current: 296352743432, replaying Lustre: 29377:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 441 previous similar messages LustreError: 29377:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=707 *** LustreError: 29375:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ================================================== 04:00:22 (1308049222) LustreError: 29376:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk =========================== 04:01:00 (1308049260) Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 04:01:28 (1308049288) Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ============================================ 04:02:14 (1308049334) Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ============== 04:03:28 (1308049408) LustreError: 29376:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms LustreError: 29376:0:(fail.c:126:__cfs_fail_timeout_set()) Skipped 3 previous similar messages LustreError: 29376:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake LustreError: 29376:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ===================== 04:04:22 (1308049462) Lustre: DEBUG MARKER: phase 2 Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks =================================================== 04:04:46 (1308049486) Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ==================================================== 04:05:52 (1308049552) Lustre: DEBUG MARKER: == replay-single test 70b: mds recovery; clients ==================================================== 04:06:18 (1308049578) Lustre: DEBUG MARKER: Started rundbench load pid=16416 ... LustreError: 1228:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 1228:0:(osd_handler.c:935:osd_ro()) Skipped 1 previous similar message Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Alloc from readonly device sdb (0x800010): [inode 3660355] [logic 0] [goal 1839618] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Alloc from readonly device sdb (0x800010): [inode 13] [logic 65] [goal 1413833] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] Alloc from readonly device sdb (0x800010): [inode 3660359] [logic 0] [goal 1838592] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Alloc from readonly device sdb (0x800010): [inode 3660361] [logic 0] [goal 1839618] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Alloc from readonly device sdb (0x800010): [inode 3660364] [logic 0] [goal 1839618] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 0] Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 26 previous similar messages LustreError: 137-5: UUID 'lustre-MDT0000_UUID' is not available for connect (stopping) LustreError: Skipped 4 previous similar messages LustreError: 29376:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-19) req@dc6a7000 x1371586166827910/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1308024402 ref 1 fl Interpret:/ffffffff/ffffffff rc -19/-1 LustreError: 29376:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 37 previous similar messages Lustre: 1368:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 1368:0:(quota_master.c:793:close_quota_files()) Skipped 7 previous similar messages Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 3 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 3 previous similar messages Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 6 previous similar messages Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775195110 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024387] [real_sent 1308024387] [current 1308024394] [deadline 7s] [delay 0s] req@c473f400 x1371559775195110/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308024394 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 3 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 3 previous similar messages LustreError: 1539:0:(mgs_handler.c:678:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS LustreError: 1539:0:(mgs_handler.c:678:mgs_handle()) Skipped 4 previous similar messages Lustre: 1539:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from 823c9d1b-d784-da93-6bef-970406ce1ace@192.168.4.15@o2ib t0 exp 00000000 cur 1308024397 last 0 Lustre: 1539:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 49 previous similar messages Lustre: 1539:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 1539:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 42 previous similar messages Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775195112 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024394] [real_sent 1308024394] [current 1308024400] [deadline 6s] [delay 0s] req@cb1acc00 x1371559775195112/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308024400 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 7 previous similar messages Lustre: 1482:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s Lustre: 1482:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message Lustre: Enabling ACL Lustre: Skipped 3 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 3 previous similar messages Lustre: 1547:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 300647711493 Lustre: 1547:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 6 previous similar messages LustreError: 1551:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 1551 LustreError: 1551:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 6 previous similar messages Lustre: 1547:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 1547:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 3 previous similar messages Lustre: 1547:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 1547:0:(mds_lov.c:1003:mds_notify()) Skipped 13 previous similar messages Lustre: 8026:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8026:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 13 previous similar messages Lustre: 8026:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000680e9b60); group 0 Lustre: 8026:0:(filter.c:2846:filter_connect()) Skipped 13 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 23 previous similar messages Lustre: 1606:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 1606:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 15 previous similar messages LustreError: 1551:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 300647711171, ql: 2, comp: 0, conn: 2, next: 300647711494, last_committed: 300647711493) Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xdad34000 -> 0xdf541e00 Lustre: 8029:0:(lustre_log.h:471:llog_group_set_export()) Skipped 29 previous similar messages Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) changing the import c5ec9800 - c3310000 Lustre: 8029:0:(llog_net.c:168:llog_receptor_accept()) Skipped 29 previous similar messages Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8029:0:(filter.c:2550:filter_llog_connect()) Skipped 15 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: Skipped 17 previous similar messages Lustre: Skipped 17 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times LustreError: 1653:0:(mdt_handler.c:2815:mdt_recovery()) operation 101 on unconnected MDS from 12345-192.168.4.18@o2ib LustreError: 1653:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 6 previous similar messages Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 2170:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 304942678929, ql: 2, comp: 0, conn: 2, next: 304942678930, last_committed: 304942678929) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775195931 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024472] [real_sent 1308024472] [current 1308024479] [deadline 7s] [delay 0s] req@ca59d400 x1371559775195931/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308024479 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 2803:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 309237646064, ql: 2, comp: 0, conn: 2, next: 309237646065, last_committed: 309237646064) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib LustreError: 3454:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 300647711171, ql: 2, comp: 0, conn: 2, next: 313532613382, last_committed: 313532613381) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775196803 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024542] [real_sent 1308024542] [current 1308024549] [deadline 7s] [delay 0s] req@dcbb0000 x1371559775196803/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308024549 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 4053:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s Lustre: 4053:0:(import.c:529:import_select_connection()) Skipped 1 previous similar message Lustre: 4123:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 300647710969, current: 300647710973, replaying Lustre: 4123:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 4160 previous similar messages LustreError: 4122:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 300647711171, ql: 2, comp: 0, conn: 2, next: 317827580616, last_committed: 317827580615) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 4762:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 300647711171, ql: 2, comp: 0, conn: 2, next: 322122547916, last_committed: 322122547915) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 5466:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 326417515269, ql: 2, comp: 0, conn: 2, next: 326417515271, last_committed: 326417515270) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775198197 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024669] [real_sent 1308024669] [current 1308024676] [deadline 7s] [delay 0s] req@f659c400 x1371559775198197/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308024676 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 6144:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 300647711171, ql: 2, comp: 0, conn: 2, next: 330712482501, last_committed: 330712482500) Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay , close ======= 04:12:23 (1308049943) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Removing read-only on unknown block (0x800010) Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775199434 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024743] [real_sent 1308024743] [current 1308024750] [deadline 7s] [delay 0s] req@e167f400 x1371559775199434/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308024750 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7270:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 1 previous similar message LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 6897:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=302 *** Lustre: 6899:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-MDT0000: 993b6001-16bd-8f2c-6a64-be226887a26f reconnecting Lustre: 6899:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 2 previous similar messages LustreError: 6897:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 335007468064, ql: 2, comp: 0, conn: 2, next: 335007468065, last_committed: 335007468064) Lustre: 6897:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ====== 04:13:26 (1308050006) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614507] [block 1427648] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614509] [block 1427136] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614508] [block 1427651] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 7674:0:(import.c:529:import_select_connection()) MGC192.168.4.128@o2ib: tried all connections, increasing latency to 6s Lustre: 7674:0:(import.c:529:import_select_connection()) Skipped 2 previous similar messages LustreError: 7742:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=30c *** LustreError: 7742:0:(ldlm_lib.c:2113:target_send_reply_msg()) @@@ dropping reply req@e4bbe42c x1371586212924844/t300647710969(300647710969) o-1->993b6001-16bd-8f2c-6a64-be226887a26f@NET_0x50000c0a80412_UUID:0/0 lens 544/544 e 0 to 0 dl 1308024847 ref 1 fl Complete:/ffffffff/ffffffff rc 301/-1 LustreError: 7742:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 339302416388, ql: 2, comp: 0, conn: 2, next: 339302416389, last_committed: 339302416388) Lustre: 7742:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: DEBUG MARKER: == replay-single test 73c: open(O_CREAT), unlink, replay, reconnect at last_replay, close ============ 04:14:30 (1308050070) Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Release to readonly device sdb (0x800010): [inode 2614507] [block 1427648] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614511] [block 1428704] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614512] [block 1428194] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614508] [block 1427651] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614510] [block 1428192] [count 2] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614510] [block 1428197] [count 1] [is_meta 1] Removing read-only on unknown block (0x800010) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775200300 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308024877] [real_sent 1308024877] [current 1308024883] [deadline 6s] [delay 0s] req@d5d99400 x1371559775200300/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308024883 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 4 previous similar messages LustreError: 8565:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 343597383684, ql: 2, comp: 0, conn: 2, next: 343597383685, last_committed: 343597383684) Lustre: 8565:0:(mdd_orphans.c:359:orph_key_test_and_del()) Found orphan! Delete it Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ==================== 04:14:48 (1308050088) Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LustreError: 23696:0:(ldlm_lockd.c:2053:ldlm_cancel_handler()) ldlm_cancel from 192.168.4.15@o2ib arrived at 1308024888 with bad export cookie 8118723492499552066 LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) LustreError: Skipped 1 previous similar message LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_connect operation failed with -19 LustreError: Skipped 2 previous similar messages LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 9514:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 9457:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 8028:0:(ldlm_lib.c:904:target_handle_connect()) lustre-OST0000: denying connection for new client 192.168.4.15@o2ib (f01d9434-7eae-4d53-e3b3-1f96da233a53): 0 clients in recovery for 69s LustreError: 8028:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 1 previous similar message LustreError: 8027:0:(ldlm_lib.c:904:target_handle_connect()) lustre-OST0000: denying connection for new client 192.168.4.15@o2ib (f01d9434-7eae-4d53-e3b3-1f96da233a53): 0 clients in recovery for 59s LustreError: 8027:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 3 previous similar messages LustreError: 8028:0:(ldlm_lib.c:904:target_handle_connect()) lustre-OST0000: denying connection for new client 192.168.4.15@o2ib (f01d9434-7eae-4d53-e3b3-1f96da233a53): 0 clients in recovery for 39s LustreError: 8028:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 7 previous similar messages LustreError: 8027:0:(ldlm_lib.c:904:target_handle_connect()) lustre-OST0000: denying connection for new client 192.168.4.15@o2ib (f01d9434-7eae-4d53-e3b3-1f96da233a53): 0 clients in recovery for 4s LustreError: 8027:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 13 previous similar messages Lustre: 9515:0:(ldlm_lib.c:1559:target_recovery_overseer()) recovery is timed out, evict stale exports LustreError: 9515:0:(genops.c:1267:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 993b6001-16bd-8f2c-6a64-be226887a26f@ LustreError: 9515:0:(genops.c:1267:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client a7954b84-49c0-eb44-edea-a997bb70078b@ Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 80a: CMD: unlink cross-node dir (fail mds with inode) ========================== 04:16:21 (1308050181) Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80b: CMD: unlink cross-node dir (fail mds with name) =========================== 04:16:21 (1308050181) Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81a: CMD: unlink cross-node file (fail mds with name) ========================== 04:16:22 (1308050182) Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 82a: CMD: mkdir cross-node dir (fail mds with inode) =========================== 04:16:23 (1308050183) Lustre: DEBUG MARKER: SKIP: replay-single test_82a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 82b: CMD: mkdir cross-node dir (fail mds with name) ============================ 04:16:23 (1308050183) Lustre: DEBUG MARKER: SKIP: replay-single test_82b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 83a: fail log_add during unlink recovery ======================================= 04:16:24 (1308050184) LustreError: 9247:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=140 *** LustreError: 9247:0:(llog_obd.c:261:llog_add()) No ctxt LustreError: 9247:0:(lov_log.c:123:lov_llog_origin_add()) Can't add llog (rc = -19) for stripe 0 Lustre: DEBUG MARKER: == replay-single test 83b: fail log_add during unlink recovery ======================================= 04:16:25 (1308050185) LustreError: 8471:0:(filter_log.c:137:filter_cancel_cookies_cb()) no valid context for group 0 LustreError: 10719:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** LustreError: 10719:0:(osd_handler.c:935:osd_ro()) Skipped 10 previous similar messages Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 72 previous similar messages Lustre: 10859:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 10859:0:(quota_master.c:793:close_quota_files()) Skipped 23 previous similar messages Release to readonly device sdb (0x800010): [inode 2614508] [block 1427651] [count 3] [is_meta 1] Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 11 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 11 previous similar messages Removing read-only on unknown block (0x800010) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 12 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 11 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 11 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 23 previous similar messages Lustre: 11031:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from d4b56168-b6b7-9abd-95b7-8869d7bbf4c8@0@lo t0 exp 00000000 cur 1308025001 last 0 Lustre: 11031:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 131 previous similar messages Lustre: 11031:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->192.168.4.128@o2ib netid 90000: select flavor null Lustre: 11031:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 139 previous similar messages Lustre: Enabling ACL Lustre: Skipped 11 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 11 previous similar messages Lustre: 11038:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 2 recoverable clients, last_transno 352187318385 Lustre: 11038:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 11 previous similar messages LustreError: 11042:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 11042 LustreError: 11042:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 11 previous similar messages Lustre: 11038:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 11038:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 11 previous similar messages Lustre: 11038:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 11038:0:(mds_lov.c:1003:mds_notify()) Skipped 35 previous similar messages Lustre: 8028:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 8028:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 34 previous similar messages Lustre: 8028:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000681b0433); group 0 Lustre: 8028:0:(filter.c:2846:filter_connect()) Skipped 40 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 65 previous similar messages Lustre: 11097:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 11097:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 25 previous similar messages LustreError: 11042:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 352187318336, ql: 2, comp: 0, conn: 2, next: 352187318386, last_committed: 352187318385) LustreError: 11042:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=140 *** LustreError: 11042:0:(llog_obd.c:261:llog_add()) No ctxt LustreError: 11042:0:(lov_log.c:123:lov_llog_origin_add()) Can't add llog (rc = -19) for stripe 0 Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ======================================= 04:16:53 (1308050213) Lustre: 11432:0:(genops.c:1378:obd_export_evict_by_uuid()) lustre-MDT0000: evicting f01d9434-7eae-4d53-e3b3-1f96da233a53 at adminstrative request LustreError: 11142:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 144 sleeping for 10000ms LustreError: 11142:0:(fail.c:126:__cfs_fail_timeout_set()) Skipped 112 previous similar messages LustreError: 11044:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-107) req@ec597400 x1371586166855541/t0(0) o-1->@:0/0 lens 192/0 e 0 to 0 dl 1308025045 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1 LustreError: 11044:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 62 previous similar messages LustreError: 11142:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 144 awake LustreError: 11142:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 103 previous similar messages Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ============= 04:17:04 (1308050224) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 11808:0:(mdt_handler.c:2815:mdt_recovery()) operation 41 on unconnected MDS from 12345-192.168.4.15@o2ib LustreError: 11808:0:(mdt_handler.c:2815:mdt_recovery()) Skipped 8 previous similar messages LustreError: 11806:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 352187318307, ql: 1, comp: 1, conn: 2, next: 356482285968, last_committed: 356482285967) Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0001: export for group 0 is changed: 0xf7d26000 -> 0xc3955e00 Lustre: 8025:0:(lustre_log.h:471:llog_group_set_export()) Skipped 75 previous similar messages Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) changing the import e167e400 - e4a28800 Lustre: 8025:0:(llog_net.c:168:llog_receptor_accept()) Skipped 75 previous similar messages Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) lustre-OST0001: Recovery from log 0x27e483/0x0:1616adbd Lustre: 8028:0:(filter.c:2550:filter_llog_connect()) Skipped 38 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans Lustre: Skipped 37 previous similar messages Lustre: Skipped 37 previous similar messages Lustre: Skipped 38 previous similar messages Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ============ 04:17:35 (1308050255) Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 4 previous similar messages Lustre: Increasing default stripe size to min 1048576 LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 12393:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 12336:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 04:18:03 (1308050283) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: Increasing default stripe size to min 1048576 Lustre: DEBUG MARKER: == replay-single test 87: write replay =============================================================== 04:18:10 (1308050290) LustreError: 13341:0:(filter.c:4564:filter_iocontrol()) *** setting device unknown-block(8,33) read-only *** Turning device sdc (0x800021) read-only Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Alloc from readonly device sdc (0x800021): [inode 122] [logic 0] [goal 28672] [ll 0] [pl 0] [lr 0] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 256] [goal 46080] [ll 255] [pl 46079] [lr 256] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 512] [goal 46592] [ll 511] [pl 46591] [lr 512] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 768] [goal 58112] [ll 767] [pl 58111] [lr 768] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 1024] [goal 58368] [ll 1023] [pl 58367] [lr 1024] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 1280] [goal 58624] [ll 1279] [pl 58623] [lr 1280] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 1536] [goal 58880] [ll 1535] [pl 58879] [lr 1536] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 122] [logic 1792] [goal 59136] [ll 1791] [pl 59135] [lr 1792] [pr 0] [len 256] [flags 32] Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Removing read-only on unknown block (0x800021) LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): recovery complete LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 13622:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 13565:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================== 04:18:30 (1308050310) LustreError: 13994:0:(filter.c:4564:filter_iocontrol()) *** setting device unknown-block(8,33) read-only *** Turning device sdc (0x800021) read-only Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Alloc from readonly device sdc (0x800021): [inode 159] [logic 0] [goal 28672] [ll 0] [pl 0] [lr 0] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 256] [goal 47104] [ll 255] [pl 47103] [lr 256] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 512] [goal 47616] [ll 511] [pl 47615] [lr 512] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 768] [goal 57600] [ll 767] [pl 57599] [lr 768] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 1024] [goal 57856] [ll 1023] [pl 57855] [lr 1024] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 1280] [goal 60672] [ll 1279] [pl 60671] [lr 1280] [pr 0] [len 256] [flags 32] Alloc from readonly device sdc (0x800021): [inode 159] [logic 1536] [goal 60928] [ll 1535] [pl 60927] [lr 1536] [pr 0] [len 256] [flags 32] Release to readonly device sdc (0x800021): [inode 159] [block 60416] [count 768] [is_meta 1] Release to readonly device sdc (0x800021): [inode 159] [block 57344] [count 512] [is_meta 1] Release to readonly device sdc (0x800021): [inode 159] [block 47360] [count 256] [is_meta 1] Release to readonly device sdc (0x800021): [inode 159] [block 46849] [count 255] [is_meta 1] Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Removing read-only on unknown block (0x800021) Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): recovery complete LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 14269:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 14212:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 14270:0:(ost_handler.c:1259:ost_brw_write()) client csum e4048ef8, server csum 60e0baeb LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: changed in transit before arrival at OST from 12345-192.168.4.15@o2ib inode [0x2000059f1:0x5:0x0] object 5762/0 extent [0-1048575] LustreError: 14270:0:(ost_handler.c:1329:ost_brw_write()) client csum e4048ef8, original server csum 60e0baeb, server csum now 60e0baeb Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ======================== 04:18:50 (1308050330) LustreError: 14604:0:(filter.c:4564:filter_iocontrol()) *** setting device unknown-block(8,33) read-only *** Turning device sdc (0x800021) read-only Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Turning device sdb (0x800010) read-only Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Alloc from readonly device sdb (0x800010): [inode 13] [logic 67] [goal 1431267] [ll 0] [pl 0] [lr 0] [pr 0] [len 1] [flags 32] Release to readonly device sdb (0x800010): [inode 2614518] [block 1432291] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614517] [block 1432288] [count 3] [is_meta 1] Release to readonly device sdb (0x800010): [inode 2614516] [block 1431776] [count 3] [is_meta 1] Removing read-only on unknown block (0x800010) Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Removing read-only on unknown block (0x800021) Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775203163 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308025139] [real_sent 1308025139] [current 1308025145] [deadline 6s] [delay 0s] req@cb93f400 x1371559775203163/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 368/392 e 0 to 1 dl 1308025145 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 5 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): recovery complete LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_connect operation failed with -19 LustreError: Skipped 3 previous similar messages Lustre: Increasing default stripe size to min 1048576 LustreError: 15454:0:(mds_lov.c:349:mds_lov_update_objids()) Unexpected gap in objids LustreError: 15454:0:(mds_lov.c:349:mds_lov_update_objids()) Skipped 1 previous similar message LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): recovery complete LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 15766:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 15709:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ================================== 04:19:28 (1308050368) Lustre: lustre-OST0000: shutting down for failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: Increasing default stripe size to min 1048576 Lustre: 16470:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Next recovery transno: 369367187511, current: 369367187537, replaying Lustre: 16470:0:(ldlm_lib.c:2019:target_queue_recovery_request()) Skipped 5307 previous similar messages LustreError: 16466:0:(ldlm_lib.c:1503:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 369367187511, ql: 1, comp: 1, conn: 2, next: 369367187537, last_committed: 369367187536) LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 16740:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect LustreError: 16669:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 8029:0:(ldlm_lib.c:904:target_handle_connect()) lustre-OST0000: denying connection for new client 192.168.4.15@o2ib (6fc37b44-f6e5-9d28-05e9-82abf82c4bf4): 0 clients in recovery for 60s Lustre: 16742:0:(ldlm_lib.c:1559:target_recovery_overseer()) recovery is timed out, evict stale exports LustreError: 16742:0:(genops.c:1267:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 03b1767b-750e-6a55-803b-62da60d6c1c1@ Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: DEBUG MARKER: == replay-single replay-single.sh test complete, duration 3924 sec =================================== 04:21:17 (1308050477) Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 14 04:21:18 PDT 2011 Lustre: DEBUG MARKER: excepting tests: Lustre: OST lustre-OST0000 has stopped. LustreError: 17488:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 17488:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc3): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-MDTffff log by user request. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: new disk, initializing LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OSTffff log by user request. Lustre: lustre-OST0000: new disk, initializing Lustre: Skipped 1 previous similar message Lustre: 18510:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled LustreError: 18431:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler Lustre: DEBUG MARKER: ost now in FULL state LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 19124:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 19124:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ============================================================ 04:21:57 (1308050517) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 3 previous similar messages LustreError: 19849:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 19849:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) ===================================== 04:22:32 (1308050552) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 20384:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 20384:0:(filter.c:1238:filter_prep_groups()) Skipped 1 previous similar message Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 1 previous similar message LustreError: 20313:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 20313:0:(obd_class.h:1593:obd_notify()) Skipped 1 previous similar message Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 5s Lustre: 7272:0:(import.c:529:import_select_connection()) Skipped 17 previous similar messages LustreError: 20723:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 20723:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) ======================================== 04:23:10 (1308050590) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 1 previous similar message LustreError: 21600:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 21600:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) ======================================== 04:23:48 (1308050628) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 22135:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 22135:0:(filter.c:1238:filter_prep_groups()) Skipped 1 previous similar message Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 1 previous similar message LustreError: 22064:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 22064:0:(obd_class.h:1593:obd_notify()) Skipped 1 previous similar message Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib LustreError: 22403:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 22403:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 04:24:27 (1308050667) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: Skipped 1 previous similar message LustreError: 23283:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 23283:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 04:25:00 (1308050700) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ======================= 04:25:46 (1308050746) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 24385:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@cd249c00 x1371559775204382/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24385:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@cd249c00 x1371559775204384/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24385:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@cd249c00 x1371559775204385/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 Lustre: 24472:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 24472:0:(filter.c:1238:filter_prep_groups()) Skipped 2 previous similar messages Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 2 previous similar messages LustreError: 24385:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 24385:0:(obd_class.h:1593:obd_notify()) Skipped 2 previous similar messages LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@f7bf2400 x1371559775204386/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@f7bf2400 x1371559775204388/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@f7bf2400 x1371559775204389/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@e83ce000 x1371559775204392/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@f7b68000 x1371559775204396/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 24471:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@f2d67800 x1371559775204398/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 24471:0:(client.c:1057:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Skipped 27 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) =================== 04:27:33 (1308050853) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 11 previous similar messages Lustre: 24916:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 90000: select flavor null Lustre: 24916:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 125 previous similar messages Lustre: 24983:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from e2f90e28-4331-e7d5-4042-db0b5ba46234@0@lo t0 exp 00000000 cur 1308025653 last 0 Lustre: 24983:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 109 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 16 previous similar messages Lustre: Enabling ACL Lustre: Skipped 11 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 10 previous similar messages Lustre: 24988:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 24988:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 11 previous similar messages Lustre: 24988:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 24988:0:(mds_lov.c:1003:mds_notify()) Skipped 21 previous similar messages Lustre: 25047:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 25047:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 49 previous similar messages LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 25488:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 25488:0:(quota_master.c:793:close_quota_files()) Skipped 25 previous similar messages LustreError: 25488:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 25488:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: MGS has stopped. Lustre: Skipped 12 previous similar messages Lustre: 25488:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775204415 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308025656] [real_sent 1308025656] [current 1308025662] [deadline 6s] [delay 0s] req@d1848400 x1371559775204415/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308025662 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 25488:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down ========================================================== 04:27:55 (1308050875) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 25710:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@dc051000 x1371559775204417/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 25710:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 1 previous similar message LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 25945:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 25945:0:(ldlm_resource.c:748:ldlm_resource_complain()) Skipped 1 previous similar message LustreError: 25945:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d76c0e80 (111542254400876/0/0/0) (rc: 0) LustreError: 25945:0:(ldlm_resource.c:754:ldlm_resource_complain()) Skipped 1 previous similar message Lustre: 25780:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000681bd331); group 0 Lustre: 25780:0:(filter.c:2846:filter_connect()) Skipped 34 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 19 previous similar messages Lustre: 25780:0:(filter.c:2550:filter_llog_connect()) lustre-OST0000: Recovery from log 0x1f/0x0:1ab52532 Lustre: 25780:0:(filter.c:2550:filter_llog_connect()) Skipped 18 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 20 previous similar messages LustreError: 26372:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 26372:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ===================================== 04:28:54 (1308050934) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: Skipped 1 previous similar message LustreError: 26671:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-11) req@ce134000 x1371590816169991/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1308025759 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1 LustreError: 26671:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 81 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ============================== 04:29:28 (1308050968) Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again =============================================== 04:29:29 (1308050969) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 28126:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 28126:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 1 previous similar message LustreError: 28126:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 LustreError: 28126:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 1 previous similar message Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup =================================================== 04:30:08 (1308051008) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 5 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup ============================================================ 04:30:41 (1308051041) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 29463:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 29463:0:(filter.c:1238:filter_prep_groups()) Skipped 5 previous similar messages Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 5 previous similar messages LustreError: 29392:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 29392:0:(obd_class.h:1593:obd_notify()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 04:31:17 (1308051077) LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 29956:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@e6c06c00 x1371559775204646/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 29956:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 30041:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@e8345c00 x1371559775204650/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ====================================================================================================== 04:31:44 (1308051104) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: Skipped 6 previous similar messages LustreError: 31131:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 31131:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 2 previous similar messages LustreError: 31131:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 LustreError: 31131:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 2 previous similar messages LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 31516:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 31516:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: dd850e80 (111542254400876/0/0/0) (rc: 0) LustreError: 31516:0:(obd_mount.c:1197:server_start_targets()) no server named lustre-MDT0000 was started LustreError: 31516:0:(obd_mount.c:1710:server_fill_super()) Unable to start targets: -6 LustreError: 31516:0:(obd_mount.c:1499:server_put_super()) no obd lustre-MDT0000 LustreError: 31516:0:(obd_mount.c:2146:lustre_fill_super()) Unable to mount (-6) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc3): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-MDTffff log by user request. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: new disk, initializing LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OSTffff log by user request. Lustre: lustre-OST0000: new disk, initializing Lustre: DEBUG MARKER: ost now in FULL state LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) LustreError: Skipped 43 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ============================================ 04:33:25 (1308051205) Lustre: DEBUG MARKER: use device /dev/sdb with MIN=2000000 LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc3): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-MDTffff log by user request. Lustre: lustre-MDT0000: new disk, initializing LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-OST0000: new disk, initializing Lustre: DEBUG MARKER: ost now in FULL state Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 5s Lustre: 7272:0:(import.c:529:import_select_connection()) Skipped 18 previous similar messages Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 76M LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc3): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc3): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-MDTffff log by user request. Lustre: Skipped 1 previous similar message Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: new disk, initializing LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-OST0000: new disk, initializing Lustre: DEBUG MARKER: ost now in FULL state Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ================================================= 04:34:48 (1308051288) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ================================================= 04:34:56 (1308051296) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 6892:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@c430fc00 x1371559775204869/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 6892:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LustreError: 6976:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@d32e6400 x1371559775204873/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ======================== 04:35:20 (1308051320) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 04:35:47 (1308051347) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: ost now in FULL state Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 04:36:10 (1308051370) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 9782:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 9633:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@e3124c00 x1371559775204945/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 9782:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d49fb080 (111542254400876/0/0/0) (rc: 0) Lustre: DEBUG MARKER: ost now in FULL state LustreError: 10375:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 10375:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 8 previous similar messages LustreError: 10375:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 LustreError: 10375:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 04:36:52 (1308051412) LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 10832:0:(ldlm_resource.c:748:ldlm_resource_complain()) Namespace MGC192.168.4.128@o2ib resource refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 10832:0:(ldlm_resource.c:754:ldlm_resource_complain()) Resource: d4c6be80 (111542254400876/0/0/0) (rc: 0) LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OSTffff log by user request. Lustre: Skipped 1 previous similar message Lustre: lustre-OST0001: new disk, initializing Lustre: DEBUG MARKER: ost2 now in FULL state Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: server umount lustre-OST0001 complete Lustre: Skipped 34 previous similar messages LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ============================================= 04:37:35 (1308051455) Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ============================== 04:37:36 (1308051456) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 16 previous similar messages Lustre: 12677:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.4.128@o2ib->MGC192.168.4.128@o2ib_0 netid 90000: select flavor null Lustre: 12677:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 96 previous similar messages Lustre: 12743:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from 17d11cf8-13aa-5d39-37d5-ebaacb693e35@0@lo t0 exp 00000000 cur 1308026255 last 0 Lustre: 12743:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 60 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 20 previous similar messages Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 4 previous similar messages Lustre: Enabling ACL Lustre: Skipped 15 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 15 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 12 previous similar messages Lustre: 12757:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 12757:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 13 previous similar messages Lustre: 12815:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 12815:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 69 previous similar messages LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OST0000 log by user request. Lustre: 13332:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 13332:0:(mds_lov.c:1003:mds_notify()) Skipped 16 previous similar messages Lustre: DEBUG MARKER: ost now in FULL state LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_statfs operation failed with -107 LustreError: Skipped 9 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: Skipped 1 previous similar message LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 0@lo. Lustre: 13929:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 13929:0:(quota_master.c:793:close_quota_files()) Skipped 31 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 16 previous similar messages Lustre: 13929:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775205103 sent from MGC192.168.4.128@o2ib to NID 0@lo has timed out for slow reply: [sent 1308026275] [real_sent 1308026275] [current 1308026281] [deadline 6s] [delay 0s] req@ced43000 x1371559775205103/t0(0) o-1->MGS@192.168.4.128@o2ib:26/25 lens 192/192 e 0 to 1 dl 1308026281 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 13929:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 04:38:14 (1308051494) LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: Skipped 5 previous similar messages Lustre: 14453:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000681bf057); group 0 Lustre: 14453:0:(filter.c:2846:filter_connect()) Skipped 26 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 16 previous similar messages Lustre: 14453:0:(filter.c:2550:filter_llog_connect()) lustre-OST0000: Recovery from log 0x1f/0x0:7a03584a Lustre: 14453:0:(filter.c:2550:filter_llog_connect()) Skipped 10 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 16 previous similar messages Lustre: Failing over lustre-MDT0000 Lustre: Skipped 34 previous similar messages Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. Lustre: Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LustreError: Skipped 5 previous similar messages LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: 14799:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 34359738368 Lustre: 14799:0:(ldlm_lib.c:1893:target_recovery_init()) Skipped 8 previous similar messages LustreError: 14803:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 14803 LustreError: 14803:0:(ldlm_lib.c:1730:target_recovery_thread()) Skipped 8 previous similar messages Lustre: 14453:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 14453:0:(ldlm_lib.c:800:target_handle_connect()) Skipped 12 previous similar messages Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 19 previous similar messages LustreError: 14805:0:(ldlm_lib.c:904:target_handle_connect()) lustre-MDT0000: denying connection for new client 192.168.4.15@o2ib (4174035d-7ea9-c9ae-39ea-e7ef7dccb431): 0 clients in recovery for 60s LustreError: 14805:0:(ldlm_lib.c:904:target_handle_connect()) Skipped 13 previous similar messages LustreError: 14966:0:(ldlm_lib.c:1852:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery Lustre: 14803:0:(ldlm_lib.c:1551:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 14803:0:(ldlm_lib.c:1551:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: Skipped 7 previous similar messages Lustre: 14453:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xf770c800 -> 0xdc7cea00 Lustre: 14453:0:(lustre_log.h:471:llog_group_set_export()) Skipped 19 previous similar messages Lustre: 14453:0:(llog_net.c:168:llog_receptor_accept()) changing the import e7f16c00 - dd93ac00 Lustre: 14453:0:(llog_net.c:168:llog_receptor_accept()) Skipped 19 previous similar messages LustreError: 14968:0:(llog_cat.c:485:llog_cat_process_thread()) llog_cat_process() failed -107 Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 04:39:17 (1308051557) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 15762:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 15762:0:(filter.c:1238:filter_prep_groups()) Skipped 16 previous similar messages Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 16 previous similar messages LustreError: 15691:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 15691:0:(obd_class.h:1593:obd_notify()) Skipped 16 previous similar messages LustreError: 15526:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-11) req@e845c800 x1371591470481417/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1308026379 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1 LustreError: 15526:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 27 previous similar messages Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 17 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 04:39:51 (1308051591) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24a mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================== 04:39:53 (1308051593) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24b mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced ================================================ 04:39:53 (1308051593) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================== 04:40:31 (1308051631) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 17554:0:(libcfs_fail.h:81:cfs_fail_check_set()) *** cfs_fail_loc=135 *** LustreError: 17554:0:(obd_config.c:519:class_setup()) setup lustre-MDT0000 failed (-2) LustreError: 17554:0:(obd_config.c:1362:class_config_llog_handler()) Err -2 on cfg command: Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f LustreError: 15c-8: MGC192.168.4.128@o2ib: The configuration from log 'lustre-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 17483:0:(obd_mount.c:1182:server_start_targets()) failed to start server lustre-MDT0000: -2 LustreError: 17483:0:(obd_mount.c:1710:server_fill_super()) Unable to start targets: -2 LustreError: 17483:0:(obd_config.c:568:class_cleanup()) Device 3 not setup LustreError: 17483:0:(obd_mount.c:2146:lustre_fill_super()) Unable to mount (-2) Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 04:40:49 (1308051649) LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 17745:0:(client.c:1046:ptlrpc_import_delay_req()) @@@ send limit expired req@e1661c00 x1371559775205266/t0(0) o-1->MGS@MGC192.168.4.128@o2ib_0:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1 LustreError: 17745:0:(client.c:1046:ptlrpc_import_delay_req()) Skipped 8 previous similar messages LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 Lustre: Skipped 4 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 04:41:43 (1308051703) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: Failing over lustre-MDT0000 Lustre: Skipped 3 previous similar messages Lustre: mdd_obd-lustre-MDT0000: shutting down for failover; client state will be preserved. LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LustreError: 19383:0:(mgs_handler.c:783:mgs_handle()) MGS handle cmd=250 rc=-19 Lustre: 19394:0:(ldlm_lib.c:1893:target_recovery_init()) RECOVERY: service lustre-MDT0000, 1 recoverable clients, last_transno 51539607552 LustreError: 19398:0:(ldlm_lib.c:1730:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 19398 Lustre: 19011:0:(ldlm_lib.c:800:target_handle_connect()) lustre-OST0000: received new MDS connection from NID 0@lo, removing former export from same NID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID Lustre: 7271:0:(mds_lov.c:1023:mds_notify()) Skipped 1 previous similar message Lustre: lustre-MDT0000: sending delayed replies to recovered clients Lustre: 19011:0:(lustre_log.h:471:llog_group_set_export()) lustre-OST0000: export for group 0 is changed: 0xf2c28600 -> 0xf4225800 Lustre: 19011:0:(lustre_log.h:471:llog_group_set_export()) Skipped 1 previous similar message Lustre: 19011:0:(llog_net.c:168:llog_receptor_accept()) changing the import c3310400 - db427800 Lustre: 19011:0:(llog_net.c:168:llog_receptor_accept()) Skipped 1 previous similar message Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting ================================================== 04:42:48 (1308051768) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (stopping) LustreError: Skipped 5 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST ==================================================== 04:43:41 (1308051821) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OST0001 log by user request. Lustre: Permanently deactivating lustre-OST0001 Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client Lustre: Skipped 2 previous similar messages Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: Permanently reactivating lustre-OST0001 Lustre: lustre-OST0001-osc-MDT0000: Connection to service lustre-OST0001 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: 21319:0:(ldlm_lib.c:606:target_handle_reconnect()) lustre-OST0001: lustre-MDT0000-mdtlov_UUID reconnecting Lustre: 21319:0:(ldlm_lib.c:606:target_handle_reconnect()) Skipped 1 previous similar message LustreError: 167-0: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: lustre-OST0001-osc-MDT0000: Connection restored to service lustre-OST0001 using nid 0@lo. LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc2): mounted filesystem with ordered data mode Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 04:44:48 (1308051888) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 3 previous similar messages LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: Regenerating lustre-OST0000 log by user request. Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 15 previous similar messages LustreError: 11-0: an error occurred while communicating with 0@lo. The obd_ping operation failed with -107 LustreError: Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 0@lo was lost; in progress operations using this service will wait for recovery to complete. Lustre: Skipped 2 previous similar messages LustreError: 24365:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 24365:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 10 previous similar messages LustreError: 24365:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 LustreError: 24365:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 10 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids ======================================================== 04:47:20 (1308052040) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: lustre-MDT0000: temporarily refusing client connection from 192.168.4.15@o2ib Lustre: Skipped 12 previous similar messages Lustre: 7272:0:(import.c:529:import_select_connection()) lustre-OST0000-osc-MDT0000: tried all connections, increasing latency to 5s Lustre: 7272:0:(import.c:529:import_select_connection()) Skipped 17 previous similar messages Lustre: 24654:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from c74ecf12-5a90-4a80-430d-25fa373e3931@192.168.4.15@o2ib t0 exp 00000000 cur 1308026858 last 0 Lustre: 24654:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 74 previous similar messages Lustre: 24654:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a8040f_UUID netid 50000: select flavor null Lustre: 24654:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 96 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Skipped 24 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 31: Connect to non-existent node (shouldn't crash) =============================== 04:49:04 (1308052144) Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade from 1.8 (not live) ================================================= 04:49:16 (1308052156) Lustre: DEBUG MARKER: SKIP: conf-sanity test_32a client only testing Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade from 1.8 with writeconf ============================================= 04:49:17 (1308052157) Lustre: DEBUG MARKER: SKIP: conf-sanity test_32b client only testing Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================= 04:49:18 (1308052158) Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount =================================================== 04:49:18 (1308052158) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode Lustre: MGS MGS started Lustre: Skipped 12 previous similar messages Lustre: MGC192.168.4.128@o2ib: Reactivating import Lustre: Skipped 15 previous similar messages Lustre: Enabling ACL Lustre: Skipped 12 previous similar messages Lustre: Enabling user_xattr Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: used disk, loading Lustre: Skipped 11 previous similar messages Lustre: 26499:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity Lustre: 26499:0:(mdt_lproc.c:254:lprocfs_wr_identity_upcall()) Skipped 11 previous similar messages Lustre: 26499:0:(mds_lov.c:1003:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID Lustre: 26499:0:(mds_lov.c:1003:mds_notify()) Skipped 12 previous similar messages Lustre: 26558:0:(debug.c:323:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 26558:0:(debug.c:323:libcfs_debug_str2mask()) Skipped 47 previous similar messages LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: 26740:0:(filter.c:1238:filter_prep_groups()) lustre-OST0000: initialize groups [0,0] Lustre: 26740:0:(filter.c:1238:filter_prep_groups()) Skipped 8 previous similar messages Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/sdc1 with recovery enabled Lustre: Skipped 8 previous similar messages LustreError: 26669:0:(obd_class.h:1593:obd_notify()) obd lustre-OST0000 has no notify handler LustreError: 26669:0:(obd_class.h:1593:obd_notify()) Skipped 8 previous similar messages LustreError: 26506:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (-11) req@d729e800 x1371592097529867/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1308026979 ref 1 fl Interpret:/ffffffff/ffffffff rc -11/-1 LustreError: 26506:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 23 previous similar messages Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1371559775206186 sent from lustre-OST0000-osc-MDT0000 to NID 0@lo has timed out for slow reply: [sent 1308026957] [real_sent 1308026957] [current 1308026962] [deadline 5s] [delay 0s] req@d071f800 x1371559775206186/t0(0) o-1->lustre-OST0000_UUID@192.168.4.128@o2ib:28/4 lens 368/392 e 0 to 1 dl 1308026962 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Lustre: 7271:0:(client.c:1775:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Lustre: 26729:0:(filter.c:2846:filter_connect()) lustre-OST0000: Received MDS connection (0x70ab8000681c0ee9); group 0 Lustre: 26729:0:(filter.c:2846:filter_connect()) Skipped 25 previous similar messages Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: Skipped 12 previous similar messages Lustre: 26728:0:(filter.c:2550:filter_llog_connect()) lustre-OST0000: Recovery from log 0x1f/0x0:7a03584a Lustre: 26728:0:(filter.c:2550:filter_llog_connect()) Skipped 10 previous similar messages Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: Skipped 12 previous similar messages Lustre: 27011:0:(quota_master.c:793:close_quota_files()) quota[0] is off already Lustre: 27011:0:(quota_master.c:793:close_quota_files()) Skipped 23 previous similar messages Lustre: MGS has stopped. Lustre: Skipped 12 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 34a: umount with opened file should be fail ====================================== 04:49:57 (1308052197) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: OST lustre-OST0000 has stopped. Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: == conf-sanity test 34b: force umount with failed mds should be normal =============================== 04:50:34 (1308052234) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 0@lo was lost; in progress operations using this service will fail. Lustre: DEBUG MARKER: == conf-sanity test 34c: force umount with failed ost should be normal =============================== 04:51:19 (1308052279) LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdb): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode LDISKFS-fs (sdc1): warning: maximal mount count reached, running e2fsck is recommended LDISKFS-fs (sdc1): mounted filesystem with ordered data mode Lustre: MGS: haven't heard from client 6d811bf9-0bee-e0de-3363-6f3aa9a0cbd4 (at 192.168.4.15@o2ib) in 55 seconds. I think it's dead, and I am evicting it. exp e2283800, cur 1308027134 expire 1308027104 last 1308027079