[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-327.36.1.el7_lustre.x86_64 (jenkins@onyx-2-sdh1-el7-x8664.onyx.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Oct 7 16:33:00 PDT 2016 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.36.1.el7_lustre.x86_64 root=UUID=d627f156-41e6-4e1a-93e8-7cbf3b92a32e ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x00000000000907ff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000090800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bb3f8fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bb3f9000-0x00000000bdd2efff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bde26fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bde27000-0x00000000bdea0fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdf2f000-0x00000000bdfaffff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdfb0000-0x00000000bdffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000cfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed19000-0x00000000fed19fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.6 present. [ 0.000000] DMI: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.99.99.x045.022820121209 02/28/2012 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x840000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CFFFF write-through [ 0.000000] D0000-D3FFF write-protect [ 0.000000] D4000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask 3FFF80000000 write-back [ 0.000000] 1 base 000080000000 mask 3FFFC0000000 write-back [ 0.000000] 2 base 000100000000 mask 3FFF00000000 write-back [ 0.000000] 3 base 000200000000 mask 3FFE00000000 write-back [ 0.000000] 4 base 000400000000 mask 3FFC00000000 write-back [ 0.000000] 5 base 000800000000 mask 3FFFC0000000 write-back [ 0.000000] 6 base 0000FF800000 mask 3FFFFF800000 write-protect [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: last_pfn = 0xbe000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fcd50-0x000fcd5f] mapped at [ffff8800000fcd50] [ 0.000000] Base memory trampoline at [ffff88000008a000] 8a000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01f3a000, 0x01f3afff] PGTABLE [ 0.000000] BRK [0x01f3b000, 0x01f3bfff] PGTABLE [ 0.000000] BRK [0x01f3c000, 0x01f3cfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x83fe00000-0x83fffffff] [ 0.000000] [mem 0x83fe00000-0x83fffffff] page 1G [ 0.000000] init_memory_mapping: [mem 0x83c000000-0x83fdfffff] [ 0.000000] [mem 0x83c000000-0x83fdfffff] page 1G [ 0.000000] init_memory_mapping: [mem 0x800000000-0x83bffffff] [ 0.000000] [mem 0x800000000-0x83bffffff] page 1G [ 0.000000] init_memory_mapping: [mem 0x00100000-0xbb3f8fff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0xbb1fffff] page 2M [ 0.000000] [mem 0xbb200000-0xbb3f8fff] page 4k [ 0.000000] init_memory_mapping: [mem 0xbdfb0000-0xbdffffff] [ 0.000000] [mem 0xbdfb0000-0xbdffffff] page 4k [ 0.000000] BRK [0x01f3d000, 0x01f3dfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x100000000-0x7ffffffff] [ 0.000000] [mem 0x100000000-0x7ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x34cb1000-0x36650fff] [ 0.000000] ACPI: RSDP 00000000000f0410 00024 (v02 INTEL ) [ 0.000000] ACPI: XSDT 00000000bdfadd98 000C4 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: FACP 00000000bdfad918 000F4 (v04 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20130517/tbfadt-653) [ 0.000000] ACPI: DSDT 00000000bdf95018 16780 (v02 INTEL S2600GL 00000099 INTL 20100331) [ 0.000000] ACPI: FACS 00000000bdfadf40 00040 [ 0.000000] ACPI: APIC 00000000bdfac718 0066A (v03 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: SPMI 00000000bdfafb98 00040 (v05 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: MCFG 00000000bdfafb18 0003C (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: WDDT 00000000bdfaff18 00040 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: SRAT 00000000bdf93c18 002A8 (v03 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: SLIT 00000000bdfafe98 00030 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: MSCT 00000000bdfaee18 00090 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: HPET 00000000bdfafe18 00038 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: SSDT 00000000bdfafd18 0002B (v02 INTEL S2600GL 00001000 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bde27018 795C4 (v02 INTEL S2600GL 00004000 INTL 20100331) [ 0.000000] ACPI: DMAR 00000000bdfad618 00148 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: SPCR 00000000bdfafd98 00050 (v01 INTEL S2600GL 06222004 INTL 20090903) [ 0.000000] ACPI: HEST 00000000bdf94f18 000A8 (v01 INTEL S2600GL 00000001 INTL 00000001) [ 0.000000] ACPI: BERT 00000000bdfafc98 00030 (v01 INTEL S2600GL 00000001 INTL 00000001) [ 0.000000] ACPI: ERST 00000000bdf94c98 00230 (v01 INTEL S2600GL 00000001 INTL 00000001) [ 0.000000] ACPI: EINJ 00000000bdfadc18 00130 (v01 INTEL S2600GL 00000001 INTL 00000001) [ 0.000000] ACPI: SSDT 00000000bdf90018 00F41 (v02 INTEL S2600GL 00000002 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdf90f98 00045 (v02 INTEL S2600GL 00000001 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdface18 00181 (v02 INTEL S2600GL 00000003 INTL 20100331) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0d -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0e -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0f -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x22 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x23 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x24 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x25 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x26 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x27 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2a -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2b -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2c -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2d -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2e -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2f -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0xbfffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x43fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x440000000-0x83fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x43fffffff] [ 0.000000] NODE_DATA [mem 0x43ffd9000-0x43fffffff] [ 0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff] [ 0.000000] NODE_DATA [mem 0x83ffd6000-0x83fffcfff] [ 0.000000] Reserving 162MB of memory at 672MB for crashkernel (System RAM: 32691MB) [ 0.000000] [ffffea0000000000-ffffea0010ffffff] PMD -> [ffff88042fe00000-ffff88043fdfffff] on node 0 [ 0.000000] [ffffea0011000000-ffffea0020ffffff] PMD -> [ffff88082f600000-ffff88083f5fffff] on node 1 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x83fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0008ffff] [ 0.000000] node 0: [mem 0x00100000-0xbb3f8fff] [ 0.000000] node 0: [mem 0xbdfb0000-0xbdffffff] [ 0.000000] node 0: [mem 0x100000000-0x43fffffff] [ 0.000000] node 1: [mem 0x440000000-0x83fffffff] [ 0.000000] On node 0 totalpages: 4174808 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3983 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 11922 pages used for memmap [ 0.000000] DMA32 zone: 762953 pages, LIFO batch:31 [ 0.000000] Normal zone: 53248 pages used for memmap [ 0.000000] Normal zone: 3407872 pages, LIFO batch:31 [ 0.000000] On node 1 totalpages: 4194304 [ 0.000000] Normal zone: 65536 pages used for memmap [ 0.000000] Normal zone: 4194304 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x0d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x0f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x27] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x2b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x2d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x2f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 128 CPUs, 96 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00090000-0x00090fff] [ 0.000000] PM: Registered nosave memory: [mem 0x00091000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbb3f9000-0xbdd2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdd2f000-0xbde26fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbde27000-0xbdea0fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdea1000-0xbdf2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdf2f000-0xbdfaffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbe000000-0xcfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xd0000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfed18fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed19000-0xfed19fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1a000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xffa1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xffa20000-0xffffffff] [ 0.000000] e820: [mem 0xd0000000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88042ee00000 s95424 r8192 d31552 u262144 [ 0.000000] pcpu-alloc: s95424 r8192 d31552 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 [ 0.000000] pcpu-alloc: [0] 016 017 018 019 020 021 022 023 [ 0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 [ 0.000000] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 [ 0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 [ 0.000000] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 [ 0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 [ 0.000000] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 [ 0.000000] pcpu-alloc: [1] 008 009 010 011 012 013 014 015 [ 0.000000] pcpu-alloc: [1] 024 025 026 027 028 029 030 031 [ 0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 [ 0.000000] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 [ 0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 [ 0.000000] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 [ 0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 [ 0.000000] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 8238321 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.36.1.el7_lustre.x86_64 root=UUID=d627f156-41e6-4e1a-93e8-7cbf3b92a32e ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Memory: 32660700k/34603008k available (6445k kernel code, 1126560k absent, 815748k reserved, 4281k data, 1632k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=2 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-127. [ 0.000000] NR_IRQS:327936 nr_irqs:2264 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 134217728 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2593.720 MHz processor [ 0.000049] Calibrating delay loop (skipped), value calculated using timer frequency.. 5187.44 BogoMIPS (lpj=2593720) [ 0.012533] pid_max: default: 131072 minimum: 1024 [ 0.018308] Security Framework initialized [ 0.023172] SELinux: Initializing. [ 0.027380] SELinux: Starting in permissive mode [ 0.029857] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes) [ 0.045494] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.056846] Mount-cache hash table entries: 4096 [ 0.062711] Initializing cgroup subsys memory [ 0.067895] Initializing cgroup subsys devices [ 0.073144] Initializing cgroup subsys freezer [ 0.078406] Initializing cgroup subsys net_cls [ 0.083652] Initializing cgroup subsys blkio [ 0.088704] Initializing cgroup subsys perf_event [ 0.094245] Initializing cgroup subsys hugetlb [ 0.099579] CPU: Physical Processor ID: 0 [ 0.104343] CPU: Processor Core ID: 0 [ 0.108719] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.123792] mce: CPU supports 20 MCE banks [ 0.128677] CPU0: Thermal monitoring enabled (TM1) [ 0.134333] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0 Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32 tlb_flushall_shift: 6 [ 0.151287] Freeing SMP alternatives: 28k freed [ 0.158056] ACPI: Core revision 20130517 [ 0.189160] ACPI: All ACPI Tables successfully acquired [ 0.199572] ftrace: allocating 24613 entries in 97 pages [ 0.218042] dmar: Host address width 46 [ 0.222610] dmar: DRHD base: 0x000000fbffe000 flags: 0x0 [ 0.228836] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe [ 0.238366] dmar: DRHD base: 0x000000ebffc000 flags: 0x1 [ 0.244590] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe [ 0.254121] dmar: RMRR base: 0x000000bdd07000 end: 0x000000bdd1dfff [ 0.261412] dmar: ATSR flags: 0x0 [ 0.265517] IOAPIC id 2 under DRHD base 0xfbffe000 IOMMU 0 [ 0.272020] IOAPIC id 0 under DRHD base 0xebffc000 IOMMU 1 [ 0.283083] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1 [ 0.289587] HPET id 0 under DRHD base 0xebffc000 [ 0.295019] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.304696] Enabled IRQ remapping in x2apic mode [ 0.310131] Enabling x2apic [ 0.313514] Enabled x2apic [ 0.316820] Switched APIC routing to cluster x2apic. [ 0.323340] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.340340] smpboot: CPU0: Genuine Intel(R) CPU @ 2.60GHz (fam: 06, model: 2d, stepping: 05) [ 0.350599] TSC deadline timer enabled [ 0.350644] Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, full-width counters, Intel PMU driver. [ 0.363215] perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode [ 0.372643] ... version: 3 [ 0.377398] ... bit width: 48 [ 0.382248] ... generic registers: 4 [ 0.387004] ... value mask: 0000ffffffffffff [ 0.393217] ... max period: 0000ffffffffffff [ 0.399430] ... fixed-purpose events: 3 [ 0.404184] ... event mask: 000000070000000f [ 0.437446] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.414898] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.531037] smpboot: Booting Node 1, Processors #8 #9 #10 #11 #12 #13 #14 #15 OK [ 0.727998] smpboot: Booting Node 0, Processors #16 #17 #18 #19 #20 #21 #22 #23 OK [ 0.844578] smpboot: Booting Node 1, Processors #24 #25 #26 #27 #28 #29 #30 #31 [ 0.963636] Brought up 32 CPUs [ 0.967797] smpboot: Total of 32 processors activated (166083.87 BogoMIPS) [ 1.031153] devtmpfs: initialized [ 1.040210] EVM: security.selinux [ 1.044189] EVM: security.ima [ 1.047777] EVM: security.capability [ 1.052196] PM: Registering ACPI NVS region [mem 0xbdd2f000-0xbde26fff] (1015808 bytes) [ 1.061648] PM: Registering ACPI NVS region [mem 0xbdea1000-0xbdf2efff] (581632 bytes) [ 1.072237] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.080389] NET: Registered protocol family 16 [ 1.085864] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.094801] ACPI: bus type PCI registered [ 1.099558] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.107144] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xc0000000-0xcfffffff] (base 0xc0000000) [ 1.118038] PCI: MMCONFIG at [mem 0xc0000000-0xcfffffff] reserved in E820 [ 1.126396] PCI: Using configuration type 1 for base access [ 1.135385] ACPI: Added _OSI(Module Device) [ 1.140339] ACPI: Added _OSI(Processor Device) [ 1.145583] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.151115] ACPI: Added _OSI(Processor Aggregator Device) [ 1.163256] ACPI: EC: Look up EC in DSDT [ 1.169225] ACPI: Executed 1 blocks of module-level executable AML code [ 1.256016] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored [ 1.271768] ACPI: Interpreter enabled [ 1.276147] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130517/hwxface-571) [ 1.287103] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130517/hwxface-571) [ 1.298060] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130517/hwxface-571) [ 1.309016] ACPI: (supports S0 S1 S5) [ 1.313386] ACPI: Using IOAPIC for interrupt routing [ 1.319250] HEST: Table parsing has been initialized. [ 1.325175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.355303] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 1.362496] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.372362] acpi PNP0A08:00: _OSC: platform does not support [AER] [ 1.379747] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.389016] acpi PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000c8000-0x000d15ff]) [ 1.403872] PCI host bridge to bus 0000:00 [ 1.408726] pci_bus 0000:00: root bus resource [bus 00-7e] [ 1.415139] pci_bus 0000:00: root bus resource [io 0x0000-0xbfff] [ 1.422330] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 1.430296] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff] [ 1.438263] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff] [ 1.446230] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff] [ 1.454195] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff] [ 1.462160] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] [ 1.470127] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] [ 1.478095] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] [ 1.486062] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] [ 1.494026] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] [ 1.501991] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff] [ 1.509958] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff] [ 1.517923] pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff] [ 1.525891] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xebffffff] [ 1.533857] pci_bus 0000:00: root bus resource [mem 0x3c0000000000-0x3c007fffffff] [ 1.542815] pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000 [ 1.542877] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold [ 1.542953] pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400 [ 1.543025] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 1.543077] pci 0000:00:01.0: System wakeup disabled by ACPI [ 1.549712] pci 0000:00:01.1: [8086:3c03] type 01 class 0x060400 [ 1.549784] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold [ 1.549835] pci 0000:00:01.1: System wakeup disabled by ACPI [ 1.556473] pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400 [ 1.556547] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 1.556598] pci 0000:00:02.0: System wakeup disabled by ACPI [ 1.563225] pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400 [ 1.563297] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold [ 1.563348] pci 0000:00:02.2: System wakeup disabled by ACPI [ 1.569984] pci 0000:00:03.0: [8086:3c08] type 01 class 0x060400 [ 1.570057] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 1.570110] pci 0000:00:03.0: System wakeup disabled by ACPI [ 1.576741] pci 0000:00:04.0: [8086:3c20] type 00 class 0x088000 [ 1.576757] pci 0000:00:04.0: reg 0x10: [mem 0xebf90000-0xebf93fff 64bit] [ 1.576876] pci 0000:00:04.1: [8086:3c21] type 00 class 0x088000 [ 1.576890] pci 0000:00:04.1: reg 0x10: [mem 0xebf80000-0xebf83fff 64bit] [ 1.577007] pci 0000:00:04.2: [8086:3c22] type 00 class 0x088000 [ 1.577021] pci 0000:00:04.2: reg 0x10: [mem 0xebf70000-0xebf73fff 64bit] [ 1.577139] pci 0000:00:04.3: [8086:3c23] type 00 class 0x088000 [ 1.577153] pci 0000:00:04.3: reg 0x10: [mem 0xebf60000-0xebf63fff 64bit] [ 1.577270] pci 0000:00:04.4: [8086:3c24] type 00 class 0x088000 [ 1.577284] pci 0000:00:04.4: reg 0x10: [mem 0xebf50000-0xebf53fff 64bit] [ 1.577401] pci 0000:00:04.5: [8086:3c25] type 00 class 0x088000 [ 1.577415] pci 0000:00:04.5: reg 0x10: [mem 0xebf40000-0xebf43fff 64bit] [ 1.577532] pci 0000:00:04.6: [8086:3c26] type 00 class 0x088000 [ 1.577546] pci 0000:00:04.6: reg 0x10: [mem 0xebf30000-0xebf33fff 64bit] [ 1.577660] pci 0000:00:04.7: [8086:3c27] type 00 class 0x088000 [ 1.577674] pci 0000:00:04.7: reg 0x10: [mem 0xebf20000-0xebf23fff 64bit] [ 1.577792] pci 0000:00:05.0: [8086:3c28] type 00 class 0x088000 [ 1.577901] pci 0000:00:05.2: [8086:3c2a] type 00 class 0x088000 [ 1.578004] pci 0000:00:05.4: [8086:3c2c] type 00 class 0x080020 [ 1.578016] pci 0000:00:05.4: reg 0x10: [mem 0xd0d60000-0xd0d60fff] [ 1.578136] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000 [ 1.578155] pci 0000:00:16.0: reg 0x10: [mem 0xd0d50000-0xd0d5000f 64bit] [ 1.578217] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 1.578280] pci 0000:00:16.1: [8086:1d3b] type 00 class 0x078000 [ 1.578299] pci 0000:00:16.1: reg 0x10: [mem 0xd0d40000-0xd0d4000f 64bit] [ 1.578360] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 1.578436] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320 [ 1.578456] pci 0000:00:1a.0: reg 0x10: [mem 0xd0d20000-0xd0d203ff] [ 1.578543] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 1.578603] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 1.585248] pci 0000:00:1c.0: [8086:1d10] type 01 class 0x060400 [ 1.585384] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 1.585440] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 1.592080] pci 0000:00:1c.7: [8086:1d1e] type 01 class 0x060400 [ 1.592161] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 1.592206] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 1.598841] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320 [ 1.598861] pci 0000:00:1d.0: reg 0x10: [mem 0xd0d10000-0xd0d103ff] [ 1.598948] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 1.599004] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 1.605636] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401 [ 1.605717] pci 0000:00:1e.0: System wakeup disabled by ACPI [ 1.612353] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100 [ 1.612515] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601 [ 1.612533] pci 0000:00:1f.2: reg 0x10: [io 0x3070-0x3077] [ 1.612540] pci 0000:00:1f.2: reg 0x14: [io 0x3060-0x3063] [ 1.612547] pci 0000:00:1f.2: reg 0x18: [io 0x3050-0x3057] [ 1.612555] pci 0000:00:1f.2: reg 0x1c: [io 0x3040-0x3043] [ 1.612562] pci 0000:00:1f.2: reg 0x20: [io 0x3020-0x303f] [ 1.612569] pci 0000:00:1f.2: reg 0x24: [mem 0xd0d00000-0xd0d007ff] [ 1.612612] pci 0000:00:1f.2: PME# supported from D3hot [ 1.612673] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500 [ 1.612687] pci 0000:00:1f.3: reg 0x10: [mem 0xebf10000-0xebf100ff 64bit] [ 1.612707] pci 0000:00:1f.3: reg 0x20: [io 0x3000-0x301f] [ 1.612834] pci 0000:01:00.0: [8086:1d74] type 01 class 0x060400 [ 1.612847] pci 0000:01:00.0: reg 0x10: [mem 0xd0a00000-0xd0a03fff] [ 1.612907] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 1.614350] pci 0000:00:01.0: PCI bridge to [bus 01-03] [ 1.620472] pci 0000:00:01.0: bridge window [io 0x2000-0x2fff] [ 1.620475] pci 0000:00:01.0: bridge window [mem 0xd0900000-0xd0afffff] [ 1.620480] pci 0000:00:01.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 1.620546] pci 0000:02:08.0: [8086:1d3f] type 01 class 0x060400 [ 1.620637] pci 0000:02:08.0: PME# supported from D0 D3hot D3cold [ 1.620701] pci 0000:01:00.0: PCI bridge to [bus 02-03] [ 1.626824] pci 0000:01:00.0: bridge window [io 0x2000-0x2fff] [ 1.626827] pci 0000:01:00.0: bridge window [mem 0xd0900000-0xd09fffff] [ 1.626833] pci 0000:01:00.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 1.626920] pci 0000:03:00.0: [8086:1d60] type 00 class 0x010700 [ 1.626939] pci 0000:03:00.0: reg 0x10: [mem 0xe9000000-0xe9007fff 64bit pref] [ 1.626952] pci 0000:03:00.0: reg 0x18: [mem 0xe8800000-0xe8ffffff 64bit pref] [ 1.626961] pci 0000:03:00.0: reg 0x20: [io 0x2100-0x21ff] [ 1.626969] pci 0000:03:00.0: reg 0x24: [io 0x2000-0x20ff] [ 1.627062] pci 0000:03:00.0: reg 0x164: [mem 0xe9010000-0xe9017fff 64bit pref] [ 1.627064] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0xe9010000-0xe9107fff 64bit pref] (contains BAR0 for 31 VFs) [ 1.639097] pci 0000:03:00.3: [8086:1d70] type 00 class 0x0c0500 [ 1.639114] pci 0000:03:00.3: reg 0x10: [mem 0xd0910000-0xd0910fff] [ 1.639148] pci 0000:03:00.3: reg 0x20: [io 0x2220-0x223f] [ 1.639215] pci 0000:03:00.3: PME# supported from D0 D3hot D3cold [ 1.639268] pci 0000:03:00.4: [8086:1d71] type 00 class 0x0c0500 [ 1.639284] pci 0000:03:00.4: reg 0x10: [mem 0xd0900000-0xd0900fff] [ 1.639319] pci 0000:03:00.4: reg 0x20: [io 0x2200-0x221f] [ 1.639384] pci 0000:03:00.4: PME# supported from D0 D3hot D3cold [ 1.639452] pci 0000:02:08.0: PCI bridge to [bus 03] [ 1.645280] pci 0000:02:08.0: bridge window [io 0x2000-0x2fff] [ 1.645284] pci 0000:02:08.0: bridge window [mem 0xd0900000-0xd09fffff] [ 1.645290] pci 0000:02:08.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 1.645380] acpiphp: Slot [2] registered [ 1.650074] pci 0000:04:00.0: [8086:1521] type 00 class 0x020000 [ 1.650086] pci 0000:04:00.0: reg 0x10: [mem 0xd0c00000-0xd0c1ffff] [ 1.650098] pci 0000:04:00.0: reg 0x18: [io 0x1000-0x101f] [ 1.650106] pci 0000:04:00.0: reg 0x1c: [mem 0xd0c20000-0xd0c23fff] [ 1.650170] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold [ 1.650199] pci 0000:04:00.0: reg 0x184: [mem 0xd0c50000-0xd0c53fff] [ 1.650200] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0xd0c50000-0xd0c6ffff] (contains BAR0 for 8 VFs) [ 1.661005] pci 0000:04:00.0: reg 0x190: [mem 0xd0c30000-0xd0c33fff] [ 1.661007] pci 0000:04:00.0: VF(n) BAR3 space: [mem 0xd0c30000-0xd0c4ffff] (contains BAR3 for 8 VFs) [ 1.673825] pci 0000:00:01.1: PCI bridge to [bus 04-05] [ 1.679950] pci 0000:00:01.1: bridge window [io 0x1000-0x1fff] [ 1.679953] pci 0000:00:01.1: bridge window [mem 0xd0c00000-0xd0cfffff] [ 1.680036] acpiphp: Slot [2-2] registered [ 1.684907] pci 0000:00:02.0: PCI bridge to [bus 06] [ 1.690818] acpiphp: Slot [2-3] registered [ 1.695689] pci 0000:00:02.2: PCI bridge to [bus 07] [ 1.701594] acpiphp: Slot [2-4] registered [ 1.706635] pci 0000:08:00.0: [15b3:673c] type 00 class 0x0c0600 [ 1.706832] pci 0000:08:00.0: reg 0x10: [mem 0xd0b00000-0xd0bfffff 64bit] [ 1.706972] pci 0000:08:00.0: reg 0x18: [mem 0xe9800000-0xe9ffffff 64bit pref] [ 1.709532] pci 0000:00:03.0: PCI bridge to [bus 08] [ 1.715362] pci 0000:00:03.0: bridge window [mem 0xd0b00000-0xd0bfffff] [ 1.715367] pci 0000:00:03.0: bridge window [mem 0xe9800000-0xe9ffffff 64bit pref] [ 1.715451] pci 0000:00:1c.0: PCI bridge to [bus 09] [ 1.721365] pci 0000:0a:00.0: [102b:0522] type 00 class 0x030000 [ 1.721393] pci 0000:0a:00.0: reg 0x10: [mem 0xea000000-0xeaffffff pref] [ 1.721409] pci 0000:0a:00.0: reg 0x14: [mem 0xd0810000-0xd0813fff] [ 1.721426] pci 0000:0a:00.0: reg 0x18: [mem 0xd0000000-0xd07fffff] [ 1.721488] pci 0000:0a:00.0: reg 0x30: [mem 0xd0800000-0xd080ffff pref] [ 1.721600] pci 0000:0a:00.0: System wakeup disabled by ACPI [ 1.730233] pci 0000:00:1c.7: PCI bridge to [bus 0a] [ 1.736079] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 1.736084] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 1.736149] pci 0000:00:1e.0: PCI bridge to [bus 0b] (subtractive decode) [ 1.744026] pci 0000:00:1e.0: bridge window [io 0x0000-0xbfff] (subtractive decode) [ 1.744027] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 1.744029] pci 0000:00:1e.0: bridge window [mem 0x000c0000-0x000c3fff] (subtractive decode) [ 1.744031] pci 0000:00:1e.0: bridge window [mem 0x000c4000-0x000c7fff] (subtractive decode) [ 1.744032] pci 0000:00:1e.0: bridge window [mem 0x000c8000-0x000cbfff] (subtractive decode) [ 1.744034] pci 0000:00:1e.0: bridge window [mem 0x000cc000-0x000cffff] (subtractive decode) [ 1.744035] pci 0000:00:1e.0: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 1.744037] pci 0000:00:1e.0: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 1.744039] pci 0000:00:1e.0: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 1.744040] pci 0000:00:1e.0: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 1.744042] pci 0000:00:1e.0: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 1.744043] pci 0000:00:1e.0: bridge window [mem 0x000e8000-0x000ebfff] (subtractive decode) [ 1.744045] pci 0000:00:1e.0: bridge window [mem 0x000ec000-0x000effff] (subtractive decode) [ 1.744046] pci 0000:00:1e.0: bridge window [mem 0x000f0000-0x000fffff] (subtractive decode) [ 1.744048] pci 0000:00:1e.0: bridge window [mem 0xd0000000-0xebffffff] (subtractive decode) [ 1.744050] pci 0000:00:1e.0: bridge window [mem 0x3c0000000000-0x3c007fffffff] (subtractive decode) [ 1.744110] pci_bus 0000:00: on NUMA node 0 [ 1.744111] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported) [ 1.753073] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.761963] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 10 11 12 14 15) [ 1.770838] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.779719] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.788595] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15) [ 1.797476] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.806357] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.815242] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.824204] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 1.831396] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.841146] acpi PNP0A08:01: _OSC: platform does not support [AER] [ 1.848446] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.857787] PCI host bridge to bus 0000:80 [ 1.862635] pci_bus 0000:80: root bus resource [bus 80-fe] [ 1.869047] pci_bus 0000:80: root bus resource [io 0x03b0-0x03df] [ 1.876234] pci_bus 0000:80: root bus resource [io 0xc000-0xffff] [ 1.883425] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff] [ 1.891391] pci_bus 0000:80: root bus resource [mem 0xec000000-0xfbffffff] [ 1.899348] pci_bus 0000:80: root bus resource [mem 0x3c0080000000-0x3c00ffffffff] [ 1.908309] pci 0000:80:01.0: [8086:3c02] type 01 class 0x060400 [ 1.908392] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold [ 1.908433] pci 0000:80:01.0: System wakeup disabled by ACPI [ 1.915071] pci 0000:80:02.0: [8086:3c04] type 01 class 0x060400 [ 1.915153] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold [ 1.915192] pci 0000:80:02.0: System wakeup disabled by ACPI [ 1.921829] pci 0000:80:03.0: [8086:3c08] type 01 class 0x060400 [ 1.921910] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold [ 1.921946] pci 0000:80:03.0: System wakeup disabled by ACPI [ 1.928583] pci 0000:80:03.2: [8086:3c0a] type 01 class 0x060400 [ 1.928664] pci 0000:80:03.2: PME# supported from D0 D3hot D3cold [ 1.928700] pci 0000:80:03.2: System wakeup disabled by ACPI [ 1.935336] pci 0000:80:04.0: [8086:3c20] type 00 class 0x088000 [ 1.935352] pci 0000:80:04.0: reg 0x10: [mem 0xfbf70000-0xfbf73fff 64bit] [ 1.935466] pci 0000:80:04.1: [8086:3c21] type 00 class 0x088000 [ 1.935481] pci 0000:80:04.1: reg 0x10: [mem 0xfbf60000-0xfbf63fff 64bit] [ 1.935591] pci 0000:80:04.2: [8086:3c22] type 00 class 0x088000 [ 1.935607] pci 0000:80:04.2: reg 0x10: [mem 0xfbf50000-0xfbf53fff 64bit] [ 1.935713] pci 0000:80:04.3: [8086:3c23] type 00 class 0x088000 [ 1.935729] pci 0000:80:04.3: reg 0x10: [mem 0xfbf40000-0xfbf43fff 64bit] [ 1.935843] pci 0000:80:04.4: [8086:3c24] type 00 class 0x088000 [ 1.935859] pci 0000:80:04.4: reg 0x10: [mem 0xfbf30000-0xfbf33fff 64bit] [ 1.935970] pci 0000:80:04.5: [8086:3c25] type 00 class 0x088000 [ 1.935985] pci 0000:80:04.5: reg 0x10: [mem 0xfbf20000-0xfbf23fff 64bit] [ 1.936095] pci 0000:80:04.6: [8086:3c26] type 00 class 0x088000 [ 1.936111] pci 0000:80:04.6: reg 0x10: [mem 0xfbf10000-0xfbf13fff 64bit] [ 1.936220] pci 0000:80:04.7: [8086:3c27] type 00 class 0x088000 [ 1.936235] pci 0000:80:04.7: reg 0x10: [mem 0xfbf00000-0xfbf03fff 64bit] [ 1.936346] pci 0000:80:05.0: [8086:3c28] type 00 class 0x088000 [ 1.936442] pci 0000:80:05.2: [8086:3c2a] type 00 class 0x088000 [ 1.936539] pci 0000:80:05.4: [8086:3c2c] type 00 class 0x080020 [ 1.936551] pci 0000:80:05.4: reg 0x10: [mem 0xec000000-0xec000fff] [ 1.936702] pci 0000:80:01.0: PCI bridge to [bus 81] [ 1.942586] pci 0000:80:02.0: PCI bridge to [bus 82] [ 1.948468] pci 0000:80:03.0: PCI bridge to [bus 83] [ 1.954343] pci 0000:80:03.2: PCI bridge to [bus 84] [ 1.960191] pci_bus 0000:80: on NUMA node 1 [ 1.960193] acpi PNP0A08:01: Disabling ASPM (FADT indicates it is unsupported) [ 1.972744] ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f]) [ 1.979645] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.989277] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 1.996996] PCI host bridge to bus 0000:7f [ 2.001853] pci_bus 0000:7f: root bus resource [bus 7f] [ 2.007981] pci 0000:7f:08.0: [8086:3c80] type 00 class 0x088000 [ 2.008031] pci 0000:7f:09.0: [8086:3c90] type 00 class 0x088000 [ 2.008075] pci 0000:7f:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.008108] pci 0000:7f:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.008144] pci 0000:7f:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.008179] pci 0000:7f:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.008214] pci 0000:7f:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.008247] pci 0000:7f:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.008280] pci 0000:7f:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.008311] pci 0000:7f:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.008346] pci 0000:7f:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.008379] pci 0000:7f:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.008412] pci 0000:7f:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.008445] pci 0000:7f:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.008478] pci 0000:7f:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.008509] pci 0000:7f:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.008542] pci 0000:7f:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.008575] pci 0000:7f:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.008610] pci 0000:7f:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.008643] pci 0000:7f:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.008679] pci 0000:7f:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.008718] pci 0000:7f:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.008766] pci 0000:7f:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.008814] pci 0000:7f:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.008861] pci 0000:7f:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.008909] pci 0000:7f:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.008957] pci 0000:7f:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.009012] pci 0000:7f:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.009052] pci 0000:7f:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.009105] pci 0000:7f:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.009154] pci 0000:7f:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.009202] pci 0000:7f:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.009250] pci 0000:7f:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.009298] pci 0000:7f:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.009349] pci 0000:7f:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.009397] pci 0000:7f:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.009446] pci 0000:7f:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.009491] pci 0000:7f:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.009529] pci 0000:7f:13.1: [8086:3c43] type 00 class 0x110100 [ 2.009562] pci 0000:7f:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.009595] pci 0000:7f:13.5: [8086:3c44] type 00 class 0x110100 [ 2.009630] pci 0000:7f:13.6: [8086:3c45] type 00 class 0x088000 [ 2.009712] ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff]) [ 2.016614] acpi PNP0A03:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.026243] acpi PNP0A03:01: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 2.033954] PCI host bridge to bus 0000:ff [ 2.038809] pci_bus 0000:ff: root bus resource [bus ff] [ 2.044937] pci 0000:ff:08.0: [8086:3c80] type 00 class 0x088000 [ 2.044984] pci 0000:ff:09.0: [8086:3c90] type 00 class 0x088000 [ 2.045032] pci 0000:ff:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.045072] pci 0000:ff:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.045112] pci 0000:ff:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.045152] pci 0000:ff:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.045193] pci 0000:ff:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.045229] pci 0000:ff:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.045272] pci 0000:ff:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.045311] pci 0000:ff:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.045351] pci 0000:ff:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.045393] pci 0000:ff:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.045436] pci 0000:ff:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.045473] pci 0000:ff:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.045512] pci 0000:ff:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.045551] pci 0000:ff:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.045591] pci 0000:ff:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.045630] pci 0000:ff:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.045671] pci 0000:ff:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.045709] pci 0000:ff:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.045755] pci 0000:ff:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.045807] pci 0000:ff:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.045867] pci 0000:ff:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.045925] pci 0000:ff:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.045989] pci 0000:ff:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.046050] pci 0000:ff:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.046108] pci 0000:ff:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.046162] pci 0000:ff:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.046208] pci 0000:ff:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.046265] pci 0000:ff:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.046322] pci 0000:ff:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.046376] pci 0000:ff:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.046435] pci 0000:ff:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.046494] pci 0000:ff:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.046555] pci 0000:ff:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.046613] pci 0000:ff:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.046667] pci 0000:ff:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.046714] pci 0000:ff:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.046752] pci 0000:ff:13.1: [8086:3c43] type 00 class 0x110100 [ 2.046793] pci 0000:ff:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.046835] pci 0000:ff:13.5: [8086:3c44] type 00 class 0x110100 [ 2.046879] pci 0000:ff:13.6: [8086:3c45] type 00 class 0x088000 [ 2.047013] ACPI: Enabled 6 GPEs in block 00 to 3F [ 2.053124] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.062681] vgaarb: loaded [ 2.065976] vgaarb: bridge control possible 0000:0a:00.0 [ 2.072281] SCSI subsystem initialized [ 2.076780] ACPI: bus type USB registered [ 2.086155] usbcore: registered new interface driver usbfs [ 2.092581] usbcore: registered new interface driver hub [ 2.098930] usbcore: registered new device driver usb [ 2.105584] PCI: Using ACPI for IRQ routing [ 2.114860] PCI: pci_cache_line_size set to 64 bytes [ 2.115073] e820: reserve RAM buffer [mem 0x00090800-0x0009ffff] [ 2.115074] e820: reserve RAM buffer [mem 0xbb3f9000-0xbbffffff] [ 2.115075] e820: reserve RAM buffer [mem 0xbe000000-0xbfffffff] [ 2.115228] NetLabel: Initializing [ 2.119303] NetLabel: domain hash size = 128 [ 2.124447] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.130284] NetLabel: unlabeled traffic allowed by default [ 2.136822] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.144724] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.153552] Switching to clocksource hpet [ 2.165086] pnp: PnP ACPI init [ 2.168787] ACPI: bus type PNP registered [ 2.173577] pnp 00:00: [dma 4] [ 2.173599] pnp 00:00: Plug and Play ACPI device, IDs PNP0200 (active) [ 2.173618] pnp 00:01: Plug and Play ACPI device, IDs INT0800 (active) [ 2.173689] pnp 00:02: Plug and Play ACPI device, IDs PNP0103 (active) [ 2.173717] pnp 00:03: Plug and Play ACPI device, IDs PNP0c04 (active) [ 2.173782] system 00:04: [io 0x0680-0x069f] has been reserved [ 2.180683] system 00:04: [io 0xffff] has been reserved [ 2.186902] system 00:04: [io 0xffff] has been reserved [ 2.193119] system 00:04: [io 0xffff] has been reserved [ 2.199334] system 00:04: [io 0x0400-0x0453] could not be reserved [ 2.206624] system 00:04: [io 0x0458-0x047f] has been reserved [ 2.213521] system 00:04: [io 0x0500-0x057f] has been reserved [ 2.220420] system 00:04: [io 0x0600-0x061f] has been reserved [ 2.227318] system 00:04: [io 0x0ca2-0x0ca5] has been reserved [ 2.234213] system 00:04: [io 0x0cf9] could not be reserved [ 2.240822] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.240851] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.240906] system 00:06: [io 0x0454-0x0457] has been reserved [ 2.247807] system 00:06: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 2.247999] pnp 00:07: [dma 0 disabled] [ 2.248043] pnp 00:07: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.248180] pnp 00:08: [dma 0 disabled] [ 2.248227] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.248520] system 00:09: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 2.256199] system 00:09: [mem 0xebfff000-0xebffffff] has been reserved [ 2.263876] system 00:09: [mem 0xc0000000-0xcfffffff] has been reserved [ 2.271551] system 00:09: [mem 0xfed20000-0xfed3ffff] has been reserved [ 2.279220] system 00:09: [mem 0xfed45000-0xfed8ffff] has been reserved [ 2.286897] system 00:09: [mem 0xff000000-0xffffffff] could not be reserved [ 2.294966] system 00:09: [mem 0xfee00000-0xfeefffff] could not be reserved [ 2.303030] system 00:09: [mem 0xfec00000-0xfecfffff] could not be reserved [ 2.311095] system 00:09: [mem 0xd0d70000-0xd0d70fff] has been reserved [ 2.318772] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.318853] system 00:0a: [mem 0xebffc000-0xebffdfff] could not be reserved [ 2.326921] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.327133] system 00:0b: [mem 0xfbffe000-0xfbffffff] could not be reserved [ 2.335203] system 00:0b: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.335284] system 00:0c: [mem 0x00000000-0x0009cfff] could not be reserved [ 2.343352] system 00:0c: Plug and Play ACPI device, IDs PNP0c01 (active) [ 2.343567] pnp: PnP ACPI: found 13 devices [ 2.348520] ACPI: bus type PNP unregistered [ 2.360854] pci 0000:02:08.0: PCI bridge to [bus 03] [ 2.366684] pci 0000:02:08.0: bridge window [io 0x2000-0x2fff] [ 2.373780] pci 0000:02:08.0: bridge window [mem 0xd0900000-0xd09fffff] [ 2.381654] pci 0000:02:08.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.390799] pci 0000:01:00.0: PCI bridge to [bus 02-03] [ 2.396922] pci 0000:01:00.0: bridge window [io 0x2000-0x2fff] [ 2.404016] pci 0000:01:00.0: bridge window [mem 0xd0900000-0xd09fffff] [ 2.411887] pci 0000:01:00.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.421031] pci 0000:00:01.0: PCI bridge to [bus 01-03] [ 2.427142] pci 0000:00:01.0: bridge window [io 0x2000-0x2fff] [ 2.434237] pci 0000:00:01.0: bridge window [mem 0xd0900000-0xd0afffff] [ 2.442106] pci 0000:00:01.0: bridge window [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.451246] pci 0000:00:01.1: PCI bridge to [bus 04-05] [ 2.457369] pci 0000:00:01.1: bridge window [io 0x1000-0x1fff] [ 2.464454] pci 0000:00:01.1: bridge window [mem 0xd0c00000-0xd0cfffff] [ 2.472328] pci 0000:00:02.0: PCI bridge to [bus 06] [ 2.478164] pci 0000:00:02.2: PCI bridge to [bus 07] [ 2.483995] pci 0000:00:03.0: PCI bridge to [bus 08] [ 2.489825] pci 0000:00:03.0: bridge window [mem 0xd0b00000-0xd0bfffff] [ 2.497697] pci 0000:00:03.0: bridge window [mem 0xe9800000-0xe9ffffff 64bit pref] [ 2.506842] pci 0000:00:1c.0: PCI bridge to [bus 09] [ 2.512684] pci 0000:00:1c.7: PCI bridge to [bus 0a] [ 2.518515] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 2.526379] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 2.535523] pci 0000:00:1e.0: PCI bridge to [bus 0b] [ 2.541358] pci_bus 0000:00: resource 4 [io 0x0000-0xbfff] [ 2.541359] pci_bus 0000:00: resource 5 [mem 0x000a0000-0x000bffff] [ 2.541361] pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff] [ 2.541363] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff] [ 2.541364] pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff] [ 2.541366] pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff] [ 2.541368] pci_bus 0000:00: resource 10 [mem 0x000d4000-0x000d7fff] [ 2.541369] pci_bus 0000:00: resource 11 [mem 0x000d8000-0x000dbfff] [ 2.541371] pci_bus 0000:00: resource 12 [mem 0x000dc000-0x000dffff] [ 2.541372] pci_bus 0000:00: resource 13 [mem 0x000e0000-0x000e3fff] [ 2.541374] pci_bus 0000:00: resource 14 [mem 0x000e4000-0x000e7fff] [ 2.541375] pci_bus 0000:00: resource 15 [mem 0x000e8000-0x000ebfff] [ 2.541377] pci_bus 0000:00: resource 16 [mem 0x000ec000-0x000effff] [ 2.541378] pci_bus 0000:00: resource 17 [mem 0x000f0000-0x000fffff] [ 2.541380] pci_bus 0000:00: resource 18 [mem 0xd0000000-0xebffffff] [ 2.541381] pci_bus 0000:00: resource 19 [mem 0x3c0000000000-0x3c007fffffff] [ 2.541383] pci_bus 0000:01: resource 0 [io 0x2000-0x2fff] [ 2.541384] pci_bus 0000:01: resource 1 [mem 0xd0900000-0xd0afffff] [ 2.541386] pci_bus 0000:01: resource 2 [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.541387] pci_bus 0000:02: resource 0 [io 0x2000-0x2fff] [ 2.541389] pci_bus 0000:02: resource 1 [mem 0xd0900000-0xd09fffff] [ 2.541391] pci_bus 0000:02: resource 2 [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.541392] pci_bus 0000:03: resource 0 [io 0x2000-0x2fff] [ 2.541394] pci_bus 0000:03: resource 1 [mem 0xd0900000-0xd09fffff] [ 2.541395] pci_bus 0000:03: resource 2 [mem 0xe8800000-0xe91fffff 64bit pref] [ 2.541397] pci_bus 0000:04: resource 0 [io 0x1000-0x1fff] [ 2.541398] pci_bus 0000:04: resource 1 [mem 0xd0c00000-0xd0cfffff] [ 2.541400] pci_bus 0000:08: resource 1 [mem 0xd0b00000-0xd0bfffff] [ 2.541402] pci_bus 0000:08: resource 2 [mem 0xe9800000-0xe9ffffff 64bit pref] [ 2.541403] pci_bus 0000:0a: resource 1 [mem 0xd0000000-0xd08fffff] [ 2.541405] pci_bus 0000:0a: resource 2 [mem 0xea000000-0xeaffffff 64bit pref] [ 2.541406] pci_bus 0000:0b: resource 4 [io 0x0000-0xbfff] [ 2.541408] pci_bus 0000:0b: resource 5 [mem 0x000a0000-0x000bffff] [ 2.541409] pci_bus 0000:0b: resource 6 [mem 0x000c0000-0x000c3fff] [ 2.541411] pci_bus 0000:0b: resource 7 [mem 0x000c4000-0x000c7fff] [ 2.541413] pci_bus 0000:0b: resource 8 [mem 0x000c8000-0x000cbfff] [ 2.541414] pci_bus 0000:0b: resource 9 [mem 0x000cc000-0x000cffff] [ 2.541416] pci_bus 0000:0b: resource 10 [mem 0x000d4000-0x000d7fff] [ 2.541417] pci_bus 0000:0b: resource 11 [mem 0x000d8000-0x000dbfff] [ 2.541419] pci_bus 0000:0b: resource 12 [mem 0x000dc000-0x000dffff] [ 2.541421] pci_bus 0000:0b: resource 13 [mem 0x000e0000-0x000e3fff] [ 2.541422] pci_bus 0000:0b: resource 14 [mem 0x000e4000-0x000e7fff] [ 2.541424] pci_bus 0000:0b: resource 15 [mem 0x000e8000-0x000ebfff] [ 2.541425] pci_bus 0000:0b: resource 16 [mem 0x000ec000-0x000effff] [ 2.541427] pci_bus 0000:0b: resource 17 [mem 0x000f0000-0x000fffff] [ 2.541428] pci_bus 0000:0b: resource 18 [mem 0xd0000000-0xebffffff] [ 2.541430] pci_bus 0000:0b: resource 19 [mem 0x3c0000000000-0x3c007fffffff] [ 2.541455] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.547292] pci 0000:80:02.0: PCI bridge to [bus 82] [ 2.553125] pci 0000:80:03.0: PCI bridge to [bus 83] [ 2.558960] pci 0000:80:03.2: PCI bridge to [bus 84] [ 2.564795] pci_bus 0000:80: resource 4 [io 0x03b0-0x03df] [ 2.564797] pci_bus 0000:80: resource 5 [io 0xc000-0xffff] [ 2.564798] pci_bus 0000:80: resource 6 [mem 0x000a0000-0x000bffff] [ 2.564800] pci_bus 0000:80: resource 7 [mem 0xec000000-0xfbffffff] [ 2.564801] pci_bus 0000:80: resource 8 [mem 0x3c0080000000-0x3c00ffffffff] [ 2.564887] NET: Registered protocol family 2 [ 2.570740] TCP established hash table entries: 262144 (order: 9, 2097152 bytes) [ 2.579933] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 2.587867] TCP: Hash tables configured (established 262144 bind 65536) [ 2.595591] TCP: reno registered [ 2.599553] UDP hash table entries: 16384 (order: 7, 524288 bytes) [ 2.606878] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes) [ 2.615027] NET: Registered protocol family 1 [ 2.661789] pci 0000:0a:00.0: Boot video device [ 2.661910] PCI: CLS 64 bytes, default 64 [ 2.661958] Unpacking initramfs... [ 3.080551] Freeing initrd memory: 26240k freed [ 3.092484] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.099983] software IO TLB [mem 0xb73f9000-0xbb3f9000] (64MB) mapped at [ffff8800b73f9000-ffff8800bb3f8fff] [ 3.113859] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer [ 3.123980] hw unit of domain pp0-core 2^-16 Joules [ 3.129716] hw unit of domain package 2^-16 Joules [ 3.135354] hw unit of domain dram 2^-16 Joules [ 3.143605] microcode: CPU0 sig=0x206d5, pf=0x1, revision=0x513 [ 3.150523] microcode: CPU1 sig=0x206d5, pf=0x1, revision=0x513 [ 3.157436] microcode: CPU2 sig=0x206d5, pf=0x1, revision=0x513 [ 3.164336] microcode: CPU3 sig=0x206d5, pf=0x1, revision=0x513 [ 3.171235] microcode: CPU4 sig=0x206d5, pf=0x1, revision=0x513 [ 3.178144] microcode: CPU5 sig=0x206d5, pf=0x1, revision=0x513 [ 3.185056] microcode: CPU6 sig=0x206d5, pf=0x1, revision=0x513 [ 3.191970] microcode: CPU7 sig=0x206d5, pf=0x1, revision=0x513 [ 3.198896] microcode: CPU8 sig=0x206d5, pf=0x1, revision=0x513 [ 3.205839] microcode: CPU9 sig=0x206d5, pf=0x1, revision=0x513 [ 3.212770] microcode: CPU10 sig=0x206d5, pf=0x1, revision=0x513 [ 3.219806] microcode: CPU11 sig=0x206d5, pf=0x1, revision=0x513 [ 3.226844] microcode: CPU12 sig=0x206d5, pf=0x1, revision=0x513 [ 3.233879] microcode: CPU13 sig=0x206d5, pf=0x1, revision=0x513 [ 3.240919] microcode: CPU14 sig=0x206d5, pf=0x1, revision=0x513 [ 3.247960] microcode: CPU15 sig=0x206d5, pf=0x1, revision=0x513 [ 3.254988] microcode: CPU16 sig=0x206d5, pf=0x1, revision=0x513 [ 3.261988] microcode: CPU17 sig=0x206d5, pf=0x1, revision=0x513 [ 3.268997] microcode: CPU18 sig=0x206d5, pf=0x1, revision=0x513 [ 3.276004] microcode: CPU19 sig=0x206d5, pf=0x1, revision=0x513 [ 3.283009] microcode: CPU20 sig=0x206d5, pf=0x1, revision=0x513 [ 3.290019] microcode: CPU21 sig=0x206d5, pf=0x1, revision=0x513 [ 3.297025] microcode: CPU22 sig=0x206d5, pf=0x1, revision=0x513 [ 3.304030] microcode: CPU23 sig=0x206d5, pf=0x1, revision=0x513 [ 3.311052] microcode: CPU24 sig=0x206d5, pf=0x1, revision=0x513 [ 3.318077] microcode: CPU25 sig=0x206d5, pf=0x1, revision=0x513 [ 3.325106] microcode: CPU26 sig=0x206d5, pf=0x1, revision=0x513 [ 3.332139] microcode: CPU27 sig=0x206d5, pf=0x1, revision=0x513 [ 3.339174] microcode: CPU28 sig=0x206d5, pf=0x1, revision=0x513 [ 3.346215] microcode: CPU29 sig=0x206d5, pf=0x1, revision=0x513 [ 3.353255] microcode: CPU30 sig=0x206d5, pf=0x1, revision=0x513 [ 3.360288] microcode: CPU31 sig=0x206d5, pf=0x1, revision=0x513 [ 3.367364] microcode: Microcode Update Driver: v2.00 , Peter Oruba [ 3.377820] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 3.384942] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 3.392815] futex hash table entries: 32768 (order: 9, 2097152 bytes) [ 3.400599] Initialise system trusted keyring [ 3.405863] audit: initializing netlink socket (disabled) [ 3.412202] type=2000 audit(1476480900.152:1): initialized [ 3.440136] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.447539] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.457066] zpool: loaded [ 3.460292] zbud: loaded [ 3.464120] VFS: Disk quotas dquot_6.5.2 [ 3.469001] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.477242] msgmni has been set to 32768 [ 3.482120] Key type big_key registered [ 3.486712] SELinux: Registering netfilter hooks [ 3.489587] NET: Registered protocol family 38 [ 3.494876] Key type asymmetric registered [ 3.499751] Asymmetric key parser 'x509' registered [ 3.505615] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 3.514591] io scheduler noop registered [ 3.519278] io scheduler deadline registered (default) [ 3.525407] io scheduler cfq registered [ 3.530731] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 3.531049] pcieport 0000:00:01.1: irq 26 for MSI/MSI-X [ 3.531372] pcieport 0000:00:02.0: irq 27 for MSI/MSI-X [ 3.531669] pcieport 0000:00:02.2: irq 28 for MSI/MSI-X [ 3.532011] pcieport 0000:00:03.0: irq 29 for MSI/MSI-X [ 3.532330] pcieport 0000:00:1c.0: irq 30 for MSI/MSI-X [ 3.532667] pcieport 0000:00:1c.7: irq 31 for MSI/MSI-X [ 3.533348] pcieport 0000:80:01.0: irq 34 for MSI/MSI-X [ 3.533593] pcieport 0000:80:02.0: irq 35 for MSI/MSI-X [ 3.533742] pcieport 0000:80:03.0: irq 36 for MSI/MSI-X [ 3.533889] pcieport 0000:80:03.2: irq 37 for MSI/MSI-X [ 3.534046] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 3.542145] pcieport 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 3.550232] pcieport 0000:02:08.0: Signaling PME through PCIe PME interrupt [ 3.558320] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 3.565923] pci 0000:03:00.3: Signaling PME through PCIe PME interrupt [ 3.573523] pci 0000:03:00.4: Signaling PME through PCIe PME interrupt [ 3.581128] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded [ 3.581157] pcieport 0000:00:01.1: Signaling PME through PCIe PME interrupt [ 3.589246] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 3.596843] pcie_pme 0000:00:01.1:pcie01: service driver pcie_pme loaded [ 3.596890] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 3.604982] pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded [ 3.605017] pcieport 0000:00:02.2: Signaling PME through PCIe PME interrupt [ 3.613104] pcie_pme 0000:00:02.2:pcie01: service driver pcie_pme loaded [ 3.613149] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 3.621242] pci 0000:08:00.0: Signaling PME through PCIe PME interrupt [ 3.628848] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded [ 3.628877] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 3.636969] pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded [ 3.637003] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt [ 3.645092] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt [ 3.652698] pcie_pme 0000:00:1c.7:pcie01: service driver pcie_pme loaded [ 3.652735] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt [ 3.660827] pcie_pme 0000:80:01.0:pcie01: service driver pcie_pme loaded [ 3.660857] pcieport 0000:80:02.0: Signaling PME through PCIe PME interrupt [ 3.668948] pcie_pme 0000:80:02.0:pcie01: service driver pcie_pme loaded [ 3.668978] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt [ 3.677070] pcie_pme 0000:80:03.0:pcie01: service driver pcie_pme loaded [ 3.677122] pcieport 0000:80:03.2: Signaling PME through PCIe PME interrupt [ 3.685215] pcie_pme 0000:80:03.2:pcie01: service driver pcie_pme loaded [ 3.685241] ioapic: probe of 0000:00:05.4 failed with error -22 [ 3.692204] ioapic: probe of 0000:80:05.4 failed with error -22 [ 3.699150] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 3.705689] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 3.713524] intel_idle: MWAIT substates: 0x21120 [ 3.713527] intel_idle: v0.4 model 0x2D [ 3.713529] intel_idle: lapic_timer_reliable_states 0xffffffff [ 3.713863] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.722621] ACPI: Power Button [PWRF] [ 3.727032] ACPI: Requesting acpi_cpufreq [ 3.768720] ERST: Error Record Serialization Table (ERST) support is initialized. [ 3.777593] pstore: Registered erst as persistent store backend [ 3.784683] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 3.793571] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.826322] 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.853584] 00:08: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.860602] Non-volatile memory driver v1.3 [ 3.865577] Linux agpgart interface v0.103 [ 3.870613] crash memory driver: version 1.1 [ 3.875788] rdac: device handler registered [ 3.880802] hp_sw: device handler registered [ 3.885872] emc: device handler registered [ 3.890747] alua: device handler registered [ 3.895753] libphy: Fixed MDIO Bus: probed [ 3.900694] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.908305] ehci-pci: EHCI PCI platform driver [ 3.913768] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 3.919969] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 3.928740] ehci-pci 0000:00:1a.0: debug port 2 [ 3.937998] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 3.938022] ehci-pci 0000:00:1a.0: irq 22, io mem 0xd0d20000 [ 3.950307] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 3.957074] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 3.964954] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.973520] usb usb1: Product: EHCI Host Controller [ 3.979258] usb usb1: Manufacturer: Linux 3.10.0-327.36.1.el7_lustre.x86_64 ehci_hcd [ 3.988405] usb usb1: SerialNumber: 0000:00:1a.0 [ 3.993943] hub 1-0:1.0: USB hub found [ 3.998425] hub 1-0:1.0: 2 ports detected [ 4.003422] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 4.009594] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 4.018362] ehci-pci 0000:00:1d.0: debug port 2 [ 4.027605] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 4.027629] ehci-pci 0000:00:1d.0: irq 20, io mem 0xd0d10000 [ 4.039353] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 4.046119] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 [ 4.054001] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.062562] usb usb2: Product: EHCI Host Controller [ 4.068299] usb usb2: Manufacturer: Linux 3.10.0-327.36.1.el7_lustre.x86_64 ehci_hcd [ 4.077445] usb usb2: SerialNumber: 0000:00:1d.0 [ 4.082970] hub 2-0:1.0: USB hub found [ 4.087451] hub 2-0:1.0: 2 ports detected [ 4.092319] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.099529] ohci-pci: OHCI PCI platform driver [ 4.104811] uhci_hcd: USB Universal Host Controller Interface driver [ 4.112272] usbcore: registered new interface driver usbserial [ 4.114418] tsc: Refined TSC clocksource calibration: 2593.749 MHz [ 4.114420] Switching to clocksource tsc [ 4.130987] usbcore: registered new interface driver usbserial_generic [ 4.138601] usbserial: USB Serial support registered for generic [ 4.145647] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 4.667195] i8042: Can't read CTR while initializing i8042 [ 4.673620] i8042: probe of i8042 failed with error -5 [ 4.679714] mousedev: PS/2 mouse device common for all mice [ 4.686353] rtc_cmos 00:05: RTC can wake from S4 [ 4.691929] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 4.699065] rtc_cmos 00:05: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 4.708169] Intel P-state driver initializing. [ 4.715337] cpuidle: using governor menu [ 4.717724] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 4.728350] hidraw: raw HID events driver (C) Jiri Kosina [ 4.734861] usbcore: registered new interface driver usbhid [ 4.741395] usbhid: USB HID core driver [ 4.746038] drop_monitor: Initializing network drop monitor service [ 4.753687] TCP: cubic registered [ 4.757688] Initializing XFRM netlink socket [ 4.762970] NET: Registered protocol family 10 [ 4.768979] NET: Registered protocol family 17 [ 4.775316] Loading compiled-in X.509 certificates [ 4.778754] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 4.788612] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.799850] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.812442] Loaded X.509 cert 'CentOS Linux kernel signing key: ea332f8e6d42202b5fe325227d4a7b47799064f1' [ 4.823672] registered taskstats version 1 [ 4.832126] usb 1-1: New USB device found, idVendor=8087, idProduct=0024 [ 4.832382] Key type trusted registered [ 4.835894] Key type encrypted registered [ 4.839138] IMA: No TPM chip found, activating TPM-bypass! [ 4.855662] rtc_cmos 00:05: setting system clock to 2016-10-14 21:35:05 UTC (1476480905) [ 4.855673] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 4.855907] hub 1-1:1.0: USB hub found [ 4.856012] hub 1-1:1.0: 6 ports detected [ 4.884297] Freeing unused kernel memory: 1632k freed [ 4.893225] usb 2-1: New USB device found, idVendor=8087, idProduct=0024 [ 4.901029] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 4.909771] hub 2-1:1.0: USB hub found [ 4.911245] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.912270] systemd[1]: Detected architecture x86-64. [ 4.912274] systemd[1]: Running in initial RAM disk. [ 4.946995] hub 2-1:1.0: 8 ports detected [ 4.961084] systemd[1]: Set hostname to . [ 5.037007] systemd[1]: Created slice -.slice. [ 5.042279] systemd[1]: Starting -.slice. [ 5.053956] systemd[1]: Listening on Journal Socket. [ 5.059833] systemd[1]: Starting Journal Socket. [ 5.072024] systemd[1]: Reached target Local File Systems. [ 5.078483] systemd[1]: Starting Local File Systems. [ 5.091020] systemd[1]: Listening on udev Control Socket. [ 5.097374] systemd[1]: Starting udev Control Socket. [ 5.110017] systemd[1]: Listening on udev Kernel Socket. [ 5.116266] systemd[1]: Starting udev Kernel Socket. [ 5.127030] systemd[1]: Reached target Swap. [ 5.132118] systemd[1]: Starting Swap. [ 5.142054] systemd[1]: Created slice System Slice. [ 5.147828] systemd[1]: Starting System Slice. [ 5.153175] systemd[1]: Started dracut ask for additional cmdline parameters. [ 5.166041] systemd[1]: Reached target Slices. [ 5.171322] systemd[1]: Starting Slices. [ 5.176181] systemd[1]: Started Load Kernel Modules. [ 5.182671] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.203737] systemd[1]: Starting dracut cmdline hook... [ 5.215591] systemd[1]: Starting Setup Virtual Console... [ 5.227170] usb 2-1.4: new full-speed USB device number 3 using ehci-pci [ 5.227761] systemd[1]: Starting Journal Service... [ 5.255438] systemd[1]: Starting Apply Kernel Variables... [ 5.272046] systemd[1]: Reached target Sockets. [ 5.277408] systemd[1]: Starting Sockets. [ 5.287050] systemd[1]: Reached target Timers. [ 5.292300] systemd[1]: Starting Timers. [ 5.302060] systemd[1]: Started Journal Service. [ 5.326859] usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10 [ 5.335057] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 5.345258] usb 2-1.4: Product: Virtual Keyboard and Mouse [ 5.353204] usb 2-1.4: Manufacturer: American Megatrends Inc. [ 5.360104] usb 2-1.4: SerialNumber: serial [ 5.361418] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input1 [ 5.412397] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0 [ 5.430997] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input2 [ 5.446518] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1 [ 5.540660] dca service started, version 1.12.1 [ 5.547381] libata version 3.00 loaded. [ 5.548415] pps_core: LinuxPPS API ver. 1 registered [ 5.555251] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 5.566703] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014) [ 5.566777] [drm] Initialized drm 1.1.0 20060810 [ 5.580839] mlx4_core: Initializing 0000:08:00.0 [ 5.588338] PTP clock support registered [ 5.595428] ahci 0000:00:1f.2: version 3.0 [ 5.595641] ahci 0000:00:1f.2: irq 39 for MSI/MSI-X [ 5.598390] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k [ 5.606227] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 5.606229] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst [ 5.616507] scsi host0: ahci [ 5.616634] scsi host1: ahci [ 5.616705] scsi host2: ahci [ 5.616763] scsi host3: ahci [ 5.616822] scsi host4: ahci [ 5.616879] scsi host5: ahci [ 5.616906] ata1: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00100 irq 39 [ 5.616908] ata2: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00180 irq 39 [ 5.616910] ata3: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00200 irq 39 [ 5.616913] ata4: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00280 irq 39 [ 5.616916] ata5: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00300 irq 39 [ 5.616919] ata6: SATA max UDMA/133 abar m2048@0xd0d00000 port 0xd0d00380 irq 39 [ 5.703351] igb: Copyright (c) 2007-2014 Intel Corporation. [ 5.704337] igb 0000:04:00.0: irq 41 for MSI/MSI-X [ 5.704352] igb 0000:04:00.0: irq 42 for MSI/MSI-X [ 5.704371] igb 0000:04:00.0: irq 43 for MSI/MSI-X [ 5.704396] igb 0000:04:00.0: irq 44 for MSI/MSI-X [ 5.704410] igb 0000:04:00.0: irq 45 for MSI/MSI-X [ 5.704425] igb 0000:04:00.0: irq 46 for MSI/MSI-X [ 5.704439] igb 0000:04:00.0: irq 47 for MSI/MSI-X [ 5.704454] igb 0000:04:00.0: irq 48 for MSI/MSI-X [ 5.704468] igb 0000:04:00.0: irq 49 for MSI/MSI-X [ 5.710693] isci: Intel(R) C600 SAS Controller Driver - version 1.1.0 [ 5.727351] isci 0000:03:00.0: driver configured for rev: 5 silicon [ 5.736190] isci 0000:03:00.0: OEM parameter table found in OROM [ 5.744741] isci 0000:03:00.0: OEM SAS parameters (version: 1.1) loaded (platform) [ 5.755624] isci 0000:03:00.0: SCU controller 0: phy 3-0 cables: {short, short, short, short} [ 5.777947] scsi host6: isci [ 5.781460] [TTM] Zone kernel: Available graphics memory: 16344300 kiB [ 5.789145] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 5.796727] [TTM] Initializing pool allocator [ 5.803909] [TTM] Initializing DMA pool allocator [ 5.809791] isci 0000:03:00.0: SCU controller 1: phy 3-0 cables: {short, short, short, short} [ 5.824884] igb 0000:04:00.0: added PHC on eth0 [ 5.830234] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection [ 5.838225] igb 0000:04:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:29:3e:55 [ 5.838299] igb 0000:04:00.0: eth0: PBA No: 100000-000 [ 5.838301] igb 0000:04:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 5.838648] scsi host7: isci [ 5.839164] isci 0000:03:00.0: irq 50 for MSI/MSI-X [ 5.839185] isci 0000:03:00.0: irq 51 for MSI/MSI-X [ 5.839199] isci 0000:03:00.0: irq 52 for MSI/MSI-X [ 5.839215] isci 0000:03:00.0: irq 53 for MSI/MSI-X [ 5.869841] fbcon: mgadrmfb (fb0) is primary device [ 5.921403] ata5: SATA link down (SStatus 0 SControl 300) [ 5.921426] ata2: SATA link down (SStatus 0 SControl 300) [ 5.921448] ata1: SATA link down (SStatus 0 SControl 300) [ 5.921487] ata3: SATA link down (SStatus 0 SControl 300) [ 5.921505] ata4: SATA link down (SStatus 0 SControl 300) [ 5.921522] ata6: SATA link down (SStatus 0 SControl 300) [ 6.103751] Console: switching to colour frame buffer device 128x48 [ 6.177082] mgag200 0000:0a:00.0: fb0: mgadrmfb frame buffer device [ 6.177083] mgag200 0000:0a:00.0: registered panic notifier [ 6.177097] [drm] Initialized mgag200 1.0.0 20110418 for 0000:0a:00.0 on minor 0 [ 7.352127] sas: phy-7:0 added to port-7:0, phy_mask:0x1 (ffffffffffffffff) [ 7.352159] sas: phy-7:1 added to port-7:1, phy_mask:0x2 (ffffffff00000000) [ 7.352172] sas: DOING DISCOVERY on port 0, pid:6 [ 7.352191] sas: DONE DISCOVERY on port 0, pid:6, result:0 [ 7.352198] sas: DOING DISCOVERY on port 1, pid:6 [ 7.352213] sas: DONE DISCOVERY on port 1, pid:6, result:0 [ 7.352286] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 7.352343] sas: ata7: end_device-7:0: dev error handler [ 7.534889] ata7.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 7.541572] ata7.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 7.569608] ata7.00: configured for UDMA/133 [ 7.574483] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 7.585641] scsi 7:0:0:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 7.594813] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 7.594876] sas: ata7: end_device-7:0: dev error handler [ 7.594897] sas: ata8: end_device-7:1: dev error handler [ 7.780168] mlx4_core 0000:08:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s [ 7.789139] mlx4_core 0000:08:00.0: PCIe link width is x8, device supports x8 [ 7.797232] mlx4_core 0000:08:00.0: irq 54 for MSI/MSI-X [ 7.797247] mlx4_core 0000:08:00.0: irq 55 for MSI/MSI-X [ 7.797261] mlx4_core 0000:08:00.0: irq 56 for MSI/MSI-X [ 7.797274] mlx4_core 0000:08:00.0: irq 57 for MSI/MSI-X [ 7.797287] mlx4_core 0000:08:00.0: irq 58 for MSI/MSI-X [ 7.797305] mlx4_core 0000:08:00.0: irq 59 for MSI/MSI-X [ 7.797319] mlx4_core 0000:08:00.0: irq 60 for MSI/MSI-X [ 7.797332] mlx4_core 0000:08:00.0: irq 61 for MSI/MSI-X [ 7.797346] mlx4_core 0000:08:00.0: irq 62 for MSI/MSI-X [ 7.797371] mlx4_core 0000:08:00.0: irq 63 for MSI/MSI-X [ 7.797384] mlx4_core 0000:08:00.0: irq 64 for MSI/MSI-X [ 7.797398] mlx4_core 0000:08:00.0: irq 65 for MSI/MSI-X [ 7.797411] mlx4_core 0000:08:00.0: irq 66 for MSI/MSI-X [ 7.797425] mlx4_core 0000:08:00.0: irq 67 for MSI/MSI-X [ 7.797438] mlx4_core 0000:08:00.0: irq 68 for MSI/MSI-X [ 7.797451] mlx4_core 0000:08:00.0: irq 69 for MSI/MSI-X [ 7.797464] mlx4_core 0000:08:00.0: irq 70 for MSI/MSI-X [ 7.797478] mlx4_core 0000:08:00.0: irq 71 for MSI/MSI-X [ 7.797491] mlx4_core 0000:08:00.0: irq 74 for MSI/MSI-X [ 7.797504] mlx4_core 0000:08:00.0: irq 75 for MSI/MSI-X [ 7.797518] mlx4_core 0000:08:00.0: irq 76 for MSI/MSI-X [ 7.797531] mlx4_core 0000:08:00.0: irq 77 for MSI/MSI-X [ 7.797544] mlx4_core 0000:08:00.0: irq 78 for MSI/MSI-X [ 7.797560] mlx4_core 0000:08:00.0: irq 79 for MSI/MSI-X [ 7.797574] mlx4_core 0000:08:00.0: irq 80 for MSI/MSI-X [ 7.797599] mlx4_core 0000:08:00.0: irq 81 for MSI/MSI-X [ 7.797612] mlx4_core 0000:08:00.0: irq 82 for MSI/MSI-X [ 7.797625] mlx4_core 0000:08:00.0: irq 83 for MSI/MSI-X [ 7.797638] mlx4_core 0000:08:00.0: irq 84 for MSI/MSI-X [ 7.797655] mlx4_core 0000:08:00.0: irq 85 for MSI/MSI-X [ 7.797668] mlx4_core 0000:08:00.0: irq 86 for MSI/MSI-X [ 7.797681] mlx4_core 0000:08:00.0: irq 87 for MSI/MSI-X [ 7.797695] mlx4_core 0000:08:00.0: irq 88 for MSI/MSI-X [ 7.797708] mlx4_core 0000:08:00.0: irq 89 for MSI/MSI-X [ 7.797721] mlx4_core 0000:08:00.0: irq 90 for MSI/MSI-X [ 7.797734] mlx4_core 0000:08:00.0: irq 91 for MSI/MSI-X [ 7.797748] mlx4_core 0000:08:00.0: irq 92 for MSI/MSI-X [ 7.797764] mlx4_core 0000:08:00.0: irq 93 for MSI/MSI-X [ 7.797777] mlx4_core 0000:08:00.0: irq 94 for MSI/MSI-X [ 7.797791] mlx4_core 0000:08:00.0: irq 95 for MSI/MSI-X [ 7.797804] mlx4_core 0000:08:00.0: irq 96 for MSI/MSI-X [ 7.797829] mlx4_core 0000:08:00.0: irq 97 for MSI/MSI-X [ 7.797842] mlx4_core 0000:08:00.0: irq 98 for MSI/MSI-X [ 7.797856] mlx4_core 0000:08:00.0: irq 99 for MSI/MSI-X [ 7.797869] mlx4_core 0000:08:00.0: irq 100 for MSI/MSI-X [ 7.797885] mlx4_core 0000:08:00.0: irq 101 for MSI/MSI-X [ 7.797899] mlx4_core 0000:08:00.0: irq 102 for MSI/MSI-X [ 7.797912] mlx4_core 0000:08:00.0: irq 103 for MSI/MSI-X [ 7.797925] mlx4_core 0000:08:00.0: irq 104 for MSI/MSI-X [ 7.797939] mlx4_core 0000:08:00.0: irq 105 for MSI/MSI-X [ 7.797952] mlx4_core 0000:08:00.0: irq 106 for MSI/MSI-X [ 7.797965] mlx4_core 0000:08:00.0: irq 107 for MSI/MSI-X [ 7.797978] mlx4_core 0000:08:00.0: irq 108 for MSI/MSI-X [ 7.797996] mlx4_core 0000:08:00.0: irq 109 for MSI/MSI-X [ 7.798009] mlx4_core 0000:08:00.0: irq 110 for MSI/MSI-X [ 7.798023] mlx4_core 0000:08:00.0: irq 111 for MSI/MSI-X [ 7.996387] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014) [ 8.004406] mlx4_en 0000:08:00.0: UDP RSS is not supported on this device [ 8.012530] mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014) [ 8.023578] mlx4_ib_add: counter index 0 for port 1 allocated 0 [ 8.031207] mlx4_ib_add: counter index 1 for port 2 allocated 0 [ 8.354772] sas: phy-6:0 added to port-6:0, phy_mask:0x1 (ffffffffffffffff) [ 8.354816] sas: phy-6:1 added to port-6:1, phy_mask:0x2 (ffffffff00000000) [ 8.354871] sas: phy-6:2 added to port-6:2, phy_mask:0x4 (ffffffff00000001) [ 8.354908] sas: phy-6:3 added to port-6:3, phy_mask:0x8 (ffffffff00000002) [ 8.354933] sas: DOING DISCOVERY on port 0, pid:453 [ 8.354966] sas: DONE DISCOVERY on port 0, pid:453, result:0 [ 8.354982] sas: DOING DISCOVERY on port 1, pid:453 [ 8.355000] sas: DONE DISCOVERY on port 1, pid:453, result:0 [ 8.355015] sas: DOING DISCOVERY on port 2, pid:453 [ 8.355056] sas: DONE DISCOVERY on port 2, pid:453, result:0 [ 8.355076] sas: DOING DISCOVERY on port 3, pid:453 [ 8.355093] sas: DONE DISCOVERY on port 3, pid:453, result:0 [ 8.355185] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 8.355205] sas: ata9: end_device-6:0: dev error handler [ 8.355630] ata8.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 8.362306] ata8.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 8.372675] ata8.00: configured for UDMA/133 [ 8.378624] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 8.379239] scsi 7:0:1:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 8.831963] ata9.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 8.840334] ata9.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 8.850984] ata9.00: configured for UDMA/133 [ 8.857552] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 8.868550] scsi 6:0:0:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 8.879494] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 8.879547] sas: ata9: end_device-6:0: dev error handler [ 8.879566] sas: ata10: end_device-6:1: dev error handler [ 9.334637] ata10.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 9.343115] ata10.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 9.353894] ata10.00: configured for UDMA/133 [ 9.360583] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 9.371856] scsi 6:0:1:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 9.382801] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 9.382879] sas: ata9: end_device-6:0: dev error handler [ 9.382898] sas: ata10: end_device-6:1: dev error handler [ 9.382906] sas: ata11: end_device-6:2: dev error handler [ 9.861046] ata11.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 9.869537] ata11.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 9.882442] ata11.00: configured for UDMA/133 [ 9.882501] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 9.893969] scsi 6:0:2:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 9.894058] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 9.894091] sas: ata9: end_device-6:0: dev error handler [ 9.894105] sas: ata10: end_device-6:1: dev error handler [ 9.894114] sas: ata11: end_device-6:2: dev error handler [ 9.894116] sas: ata12: end_device-6:3: dev error handler [ 10.378684] ata12.00: ATA-8: ST9500423AS, 0001SDM5, max UDMA/133 [ 10.387219] ata12.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 10.398051] ata12.00: configured for UDMA/133 [ 10.404647] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 10.415369] scsi 6:0:3:0: Direct-Access ATA ST9500423AS SDM5 PQ: 0 ANSI: 5 [ 10.437125] sd 6:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437235] sd 6:0:2:0: [sdc] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437238] sd 6:0:2:0: [sdc] 4096-byte physical blocks [ 10.437286] sd 6:0:2:0: [sdc] Write Protect is off [ 10.437289] sd 6:0:2:0: [sdc] Mode Sense: 00 3a 00 00 [ 10.437291] sd 6:0:1:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437294] sd 6:0:1:0: [sdb] 4096-byte physical blocks [ 10.437320] sd 6:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.437326] sd 6:0:3:0: [sdd] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437329] sd 6:0:3:0: [sdd] 4096-byte physical blocks [ 10.437362] sd 7:0:0:0: [sde] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437366] sd 7:0:0:0: [sde] 4096-byte physical blocks [ 10.437383] sd 7:0:1:0: [sdf] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 10.437385] sd 7:0:1:0: [sdf] 4096-byte physical blocks [ 10.437435] sd 7:0:1:0: [sdf] Write Protect is off [ 10.437437] sd 7:0:1:0: [sdf] Mode Sense: 00 3a 00 00 [ 10.437441] sd 7:0:0:0: [sde] Write Protect is off [ 10.437444] sd 7:0:0:0: [sde] Mode Sense: 00 3a 00 00 [ 10.437459] sd 7:0:1:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.437462] sd 6:0:1:0: [sdb] Write Protect is off [ 10.437465] sd 6:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [ 10.437476] sd 7:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.437510] sd 6:0:3:0: [sdd] Write Protect is off [ 10.437514] sd 6:0:3:0: [sdd] Mode Sense: 00 3a 00 00 [ 10.437527] sd 6:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.437566] sd 6:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.456734] sdd: [ 10.457370] sd 6:0:3:0: [sdd] Attached SCSI disk [ 10.482321] sdf: [ 10.482501] sd 7:0:1:0: [sdf] Attached SCSI disk [ 10.482594] sdb: sdb1 [ 10.482873] sd 6:0:1:0: [sdb] Attached SCSI disk [ 10.485196] sde: [ 10.485366] sd 7:0:0:0: [sde] Attached SCSI disk [ 10.500095] sdc: [ 10.500268] sd 6:0:2:0: [sdc] Attached SCSI disk [ 10.676830] sd 6:0:0:0: [sda] 4096-byte physical blocks [ 10.684510] sd 6:0:0:0: [sda] Write Protect is off [ 10.691612] sd 6:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 10.691662] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.754056] sda: sda1 sda2 [ 10.759404] sd 6:0:0:0: [sda] Attached SCSI disk [ 11.476436] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [ 11.506087] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 12.370342] systemd-journald[546]: Received SIGTERM from PID 1 (systemd). [ 13.152035] SELinux: Disabled at runtime. [ 13.158101] SELinux: Unregistering netfilter hooks [ 13.177232] type=1404 audit(1476480913.832:2): selinux=0 auid=4294967295 ses=4294967295 [ 13.447696] ip_tables: (C) 2000-2006 Netfilter Core Team [ 13.455317] systemd[1]: Inserted module 'ip_tables' [ 15.283277] EXT4-fs (sda1): re-mounted. Opts: (null) [ 16.022524] systemd-journald[896]: Received request to flush runtime journal from PID 1 [ 17.356130] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 17.417896] ipmi message handler version 39.2 [ 17.488926] EDAC MC: Ver: 3.0.0 [ 17.516158] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 17.524322] mei_me 0000:00:16.0: initialization failed. [ 17.537457] wmi: Mapper loaded [ 17.557482] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 17.566599] ACPI Warning: SystemIO range 0x0000000000003000-0x000000000000301f conflicts with OpRegion 0x0000000000003000-0x000000000000300f (\_SB_.PCI0.SBUS.SMBI) (20130517/utaddress-254) [ 17.587062] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 17.599444] i801_smbus 0000:03:00.3: SMBus using PCI interrupt [ 17.606898] i801_smbus 0000:03:00.4: SMBus using PCI interrupt [ 17.624121] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 17.624128] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 17.624133] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 17.624135] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 17.624140] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 17.624144] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 17.624146] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 17.624150] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 17.624154] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 17.624156] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 17.624160] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 17.624164] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 17.624166] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 17.624171] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 17.624175] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 17.624176] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 17.624181] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 17.624185] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 17.624186] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 17.624191] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 17.624195] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 17.624196] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 17.624201] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 17.624205] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 17.624206] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 17.624211] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 17.624214] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 17.624216] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 17.624220] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 17.624224] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 17.624226] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 17.624230] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 17.624234] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 17.624345] EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0 [ 17.635615] EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0 [ 17.646768] EDAC sbridge: Ver: 1.1.1 [ 17.658073] iTCO_vendor_support: vendor-support=0 [ 17.666237] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 17.673341] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 17.716210] ioatdma: Intel(R) QuickData Technology Driver 4.00 [ 17.723923] ioatdma 0000:00:04.0: irq 114 for MSI/MSI-X [ 17.724145] [Firmware Warn]: ioatdma 0000:00:04.0: APICID_TAG_MAP set incorrectly by BIOS, disabling DCA [ 17.735909] ioatdma 0000:00:04.1: irq 116 for MSI/MSI-X [ 17.736285] ioatdma 0000:00:04.2: irq 117 for MSI/MSI-X [ 17.736609] ioatdma 0000:00:04.3: irq 118 for MSI/MSI-X [ 17.736965] ioatdma 0000:00:04.4: irq 119 for MSI/MSI-X [ 17.737340] ioatdma 0000:00:04.5: irq 120 for MSI/MSI-X [ 17.737717] ioatdma 0000:00:04.6: irq 121 for MSI/MSI-X [ 17.738036] ioatdma 0000:00:04.7: irq 122 for MSI/MSI-X [ 17.738554] ioatdma 0000:80:04.0: irq 124 for MSI/MSI-X [ 17.738922] ioatdma 0000:80:04.1: irq 126 for MSI/MSI-X [ 17.739177] ioatdma 0000:80:04.2: irq 127 for MSI/MSI-X [ 17.739457] ioatdma 0000:80:04.3: irq 128 for MSI/MSI-X [ 17.739924] ioatdma 0000:80:04.4: irq 129 for MSI/MSI-X [ 17.740407] ioatdma 0000:80:04.5: irq 130 for MSI/MSI-X [ 17.740922] ioatdma 0000:80:04.6: irq 131 for MSI/MSI-X [ 17.741404] ioatdma 0000:80:04.7: irq 132 for MSI/MSI-X [ 17.985631] sd 6:0:0:0: Attached scsi generic sg0 type 0 [ 17.993271] sd 6:0:1:0: Attached scsi generic sg1 type 0 [ 18.000876] sd 6:0:2:0: Attached scsi generic sg2 type 0 [ 18.008226] sd 6:0:3:0: Attached scsi generic sg3 type 0 [ 18.015565] sd 7:0:0:0: Attached scsi generic sg4 type 0 [ 18.022809] sd 7:0:1:0: Attached scsi generic sg5 type 0 [ 18.029789] IPMI System Interface driver. [ 18.035176] ipmi_si: probing via SMBIOS [ 18.040327] ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 18.040329] ipmi_si: Adding SMBIOS-specified kcs state machine [ 18.040331] ipmi_si: probing via SPMI [ 18.040332] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0 [ 18.040333] ipmi_si: Adding SPMI-specified kcs state machine duplicate interface [ 18.040334] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 18.152808] ipmi_si ipmi_si.0: Found new BMC (man_id: 0x000157, prod_id: 0x0049, dev_id: 0x21) [ 18.163628] ipmi_si ipmi_si.0: IPMI kcs interface initialized [ 18.284328] ipmi device interface [ 18.325892] IPMI SSIF Interface driver [ 19.339906] Adding 16465916k swap on /dev/sda2. Priority:-1 extents:1 across:16465916k FS [ 19.906312] type=1305 audit(1476480920.557:3): audit_pid=1089 old=0 auid=4294967295 ses=4294967295 res=1 [ 22.038379] AES CTR mode by8 optimization enabled [ 22.049669] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 22.077772] alg: No test for crc32 (crc32-pclmul) [ 23.711010] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 23.792367] intel_rapl: Found RAPL domain package [ 23.798658] intel_rapl: Found RAPL domain core [ 23.804589] intel_rapl: Found RAPL domain package [ 23.810748] intel_rapl: Found RAPL domain core [ 23.857695] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 24.773786] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 24.782617] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 24.794384] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 24.803396] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 24.994433] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 25.092842] Loading iSCSI transport class v2.0-870. [ 25.144039] iscsi: registered transport (iser) [ 25.290671] RPC: Registered named UNIX socket transport module. [ 25.290672] RPC: Registered udp transport module. [ 25.290673] RPC: Registered tcp transport module. [ 25.290675] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 25.341026] RPC: Registered rdma transport module. [ 27.116431] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 27.141710] igb 0000:04:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 27.152307] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 27.171940] ib0: enabling connected mode will cause multicast packet drops [ 27.179747] ib0: mtu > 2044 will cause multicast packet drops. [ 224.796010] FS-Cache: Loaded [ 224.852425] FS-Cache: Netfs 'nfs' registered for caching [ 224.871121] Key type dns_resolver registered [ 224.897081] NFS: Registering the id_resolver key type [ 224.902784] Key type id_resolver registered [ 224.907502] Key type id_legacy registered [ 434.869662] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 434.884342] LNet: HW CPU cores: 32, npartitions: 2 [ 434.894039] alg: No test for adler32 (adler32-zlib) [ 434.899839] alg: No test for crc32 (crc32-table) [ 442.935314] Lustre: Lustre: Build Version: 2.8.59 [ 442.973301] LNet: Added LNI 10.2.4.56@tcp [8/256/0/180] [ 442.979322] LNet: Accept secure, port 988 [ 443.049940] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 443.080825] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,,errors=remount-ro,no_mbcache,nodelalloc [ 443.748926] LustreError: 23157:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 444.190903] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 450.429160] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 450.439841] Lustre: lustre-OST0000: Connection restored to cca56237-4d15-8d3b-2ad8-30f0efd5b620 (at 10.2.4.37@tcp) [ 451.564671] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) [ 496.930370] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 497.403998] Lustre: lustre-OST0000: Recovery over after 0:47, of 3 clients 3 recovered and 0 were evicted. [ 497.434808] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4 to 0x0:33 [ 780.465874] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 14:48:00 PDT 2016 [ 783.791995] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 786.071818] Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b [ 786.431596] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o [ 786.926671] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 14:48:06 (1476481686) [ 788.038120] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 14:48:07 (1476481687) [ 789.145308] Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 14:48:09 (1476481689) [ 790.298112] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 14:48:10 (1476481690) [ 791.456253] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ====================== 14:48:11 (1476481691) [ 792.604939] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ======================= 14:48:12 (1476481692) [ 793.800799] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) == 14:48:13 (1476481693) [ 794.962439] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============== 14:48:14 (1476481694) [ 796.253481] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ==== 14:48:16 (1476481696) [ 797.361512] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ==== 14:48:17 (1476481697) [ 798.538895] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ==== 14:48:18 (1476481698) [ 799.725390] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? == 14:48:19 (1476481699) [ 801.023365] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) == 14:48:20 (1476481700) [ 802.208449] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ====== 14:48:22 (1476481702) [ 803.364390] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ======= 14:48:23 (1476481703) [ 804.540823] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ========= 14:48:24 (1476481704) [ 805.686536] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================== 14:48:25 (1476481705) [ 806.856731] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ======== 14:48:26 (1476481706) [ 808.006636] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============== 14:48:27 (1476481707) [ 809.310800] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ========== 14:48:29 (1476481709) [ 810.483099] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ========== 14:48:30 (1476481710) [ 811.674395] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ========= 14:48:31 (1476481711) [ 812.859396] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ============ 14:48:32 (1476481712) [ 813.991817] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f =================== 14:48:33 (1476481713) [ 815.173675] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ==================== 14:48:35 (1476481715) [ 816.355062] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ================ 14:48:36 (1476481716) [ 817.530557] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ====== 14:48:37 (1476481717) [ 818.653218] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ========================== 14:48:38 (1476481718) [ 819.811031] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ====== 14:48:39 (1476481719) [ 820.976800] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================== 14:48:40 (1476481720) [ 822.192531] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries == 14:48:42 (1476481722) [ 824.287847] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug == 14:48:44 (1476481724) [ 825.699019] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink == 14:48:45 (1476481725) [ 827.118988] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled =========================== 14:48:47 (1476481727) [ 828.348261] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========== 14:48:48 (1476481728) [ 829.496864] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink == 14:48:49 (1476481729) [ 844.102929] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 844.118695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 851.105894] Lustre: 23241:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476481744/real 1476481744] req@ffff8803f59d8c00 x1548202900718512/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476481751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 851.148312] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 851.805665] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) [ 857.173922] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476481751/real 1476481751] req@ffff8803f59d9200 x1548202900718560/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476481757 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 859.915205] Lustre: lustre-OST0000: deleting orphan objects from 0x0:57 to 0x0:97 [ 861.000523] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir == 14:49:20 (1476481760) [ 861.892781] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [ 863.123322] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 14:49:23 (1476481763) [ 883.667435] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) [ 886.234439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 887.189351] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476481776/real 1476481776] req@ffff8803f59d9500 x1548202900718576/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476481787 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 887.668298] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 889.425031] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ================================ 14:49:49 (1476481789) [ 890.592078] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ============= 14:49:50 (1476481790) [ 891.855853] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ================ 14:49:51 (1476481791) [ 892.991165] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ==== 14:49:52 (1476481792) [ 894.141276] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================ 14:49:54 (1476481794) [ 895.279621] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... ============================= 14:49:55 (1476481795) [ 895.823894] Lustre: DEBUG MARKER: 1 done [ 896.217809] Lustre: DEBUG MARKER: 2 done [ 896.600931] Lustre: DEBUG MARKER: 3 done [ 897.701564] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================== 14:49:57 (1476481797) [ 899.020753] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user ================= 14:49:58 (1476481798) [ 900.356751] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============================ 14:50:00 (1476481800) [ 901.197668] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 901.220554] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc5443dbe to 0xedc7dcbfc5461a0b [ 901.221997] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 901.564694] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ============================ 14:50:01 (1476481801) [ 902.724830] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target == 14:50:02 (1476481802) [ 904.542455] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target == 14:50:04 (1476481804) [ 905.715023] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target == 14:50:05 (1476481805) [ 906.853551] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 14:50:06 (1476481806) [ 908.016251] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ======== 14:50:07 (1476481807) [ 909.209862] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ====== 14:50:09 (1476481809) [ 909.923433] LustreError: 23291:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 910.409407] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ======== 14:50:10 (1476481810) [ 911.624175] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e == 14:50:11 (1476481811) [ 912.805655] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a == 14:50:12 (1476481812) [ 913.925916] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================== 14:50:13 (1476481813) [ 915.105031] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ========= 14:50:14 (1476481814) [ 916.247413] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ========================== 14:50:16 (1476481816) [ 917.405631] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =========== 14:50:17 (1476481817) [ 918.693666] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) == 14:50:18 (1476481818) [ 920.086398] Lustre: DEBUG MARKER: SKIP: sanity test_24o skipping SLOW test 24o [ 920.473010] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b == 14:50:20 (1476481820) [ 921.664326] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ===== 14:50:21 (1476481821) [ 922.787146] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ======= 14:50:22 (1476481822) [ 923.959764] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c === 14:50:23 (1476481823) [ 925.135287] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a === 14:50:25 (1476481825) [ 926.376332] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 14:50:26 (1476481826) [ 927.650054] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) == 14:50:27 (1476481827) [ 1358.582042] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb == 14:57:38 (1476482258) [ 1359.828041] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link == 14:57:39 (1476482259) [ 1360.226780] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 1361.400083] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed == 14:57:41 (1476482261) [ 1361.800000] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 1362.979133] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. == 14:57:42 (1476482262) [ 1382.804873] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries == 14:58:02 (1476482282) [ 1383.216059] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 1384.350207] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir == 14:58:04 (1476482284) [ 1384.741632] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 1385.853928] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [ 1386.239665] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link == 14:58:05 (1476482285) [ 1386.634741] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 1387.801795] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ================= 14:58:07 (1476482287) [ 1388.993239] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ================= 14:58:08 (1476482288) [ 1390.194260] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ========================= 14:58:09 (1476482289) [ 1391.378434] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ======== 14:58:11 (1476482291) [ 1392.583928] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================== 14:58:12 (1476482292) [ 1393.790186] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ======== 14:58:13 (1476482293) [ 1394.924269] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ======== 14:58:14 (1476482294) [ 1396.062974] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink === 14:58:15 (1476482295) [ 1397.284454] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ==================================== 14:58:16 (1476482296) [ 1397.728413] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [ 1398.903356] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 14:58:18 (1476482298) [ 1399.303105] Lustre: DEBUG MARKER: SKIP: sanity test_27b skipping 2-stripe test [ 1400.480180] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================== 14:58:20 (1476482300) [ 1401.678115] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ======== 14:58:21 (1476482301) [ 1403.114685] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) == 14:58:22 (1476482302) [ 1404.255797] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects == 14:58:23 (1476482303) [ 1405.442119] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects == 14:58:25 (1476482305) [ 1406.627614] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) == 14:58:26 (1476482306) [ 1407.831983] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps =============== 14:58:27 (1476482307) [ 1409.055116] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) == 14:58:28 (1476482308) [ 1410.226252] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 1410.638336] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ==================== 14:58:30 (1476482310) [ 1411.039764] Lustre: DEBUG MARKER: SKIP: sanity test_27n too few OSTs [ 1412.194415] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ====== 14:58:31 (1476482311) [ 1412.592103] Lustre: DEBUG MARKER: SKIP: sanity test_27o too few OSTs [ 1413.737495] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ===== 14:58:33 (1476482313) [ 1414.136281] Lustre: DEBUG MARKER: SKIP: sanity test_27p too few OSTs [ 1415.286363] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ===== 14:58:34 (1476482314) [ 1415.655846] Lustre: DEBUG MARKER: SKIP: sanity test_27q too few OSTs [ 1416.820995] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) === 14:58:36 (1476482316) [ 1417.219525] Lustre: DEBUG MARKER: SKIP: sanity test_27r too few OSTs [ 1418.342198] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) == 14:58:37 (1476482317) [ 1419.480321] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly == 14:58:39 (1476482319) [ 1420.648438] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ============ 14:58:40 (1476482320) [ 1421.030149] Lustre: DEBUG MARKER: SKIP: sanity test_27u too few OSTs [ 1422.196988] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST =================== 14:58:41 (1476482321) [ 1422.582807] Lustre: DEBUG MARKER: SKIP: sanity test_27v too few OSTs [ 1423.751806] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option == 14:58:43 (1476482323) [ 1424.965292] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options == 14:58:44 (1476482324) [ 1425.337566] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [ 1426.478627] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 14:58:46 (1476482326) [ 1426.882705] Lustre: DEBUG MARKER: SKIP: sanity test_27x 1 < 2 OSTs [ 1428.024038] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive == 14:58:47 (1476482327) [ 1428.416289] Lustre: DEBUG MARKER: SKIP: sanity test_27y 1 < 2 OSTs -- skipping [ 1429.612647] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems == 14:58:49 (1476482329) [ 1435.182302] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 1435.601112] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1a0fd ver 0x0 [ 1435.984796] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1a0fd, count: 1 [ 1436.377374] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:165/0xa5 seq:0 [ 1437.057990] Lustre: Failing over lustre-OST0000 [ 1437.615024] Lustre: server umount lustre-OST0000 complete [ 1438.658948] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 1438.689490] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [ 1440.637935] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 1440.652807] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 1441.096689] LustreError: 38245:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 1441.464196] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 1444.232575] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 1444.246156] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 1444.260622] Lustre: Skipped 1 previous similar message [ 1446.878718] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 1446.901166] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1447.391063] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 1447.778644] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1a0ff ver 0x0 [ 1448.158352] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1a0ff, count: 1 [ 1448.545547] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:166/0xa6 seq:0 [ 1449.224003] Lustre: Failing over lustre-OST0000 [ 1450.486186] Lustre: server umount lustre-OST0000 complete [ 1451.507674] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 1451.535611] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [ 1453.319060] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 1453.339916] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 1453.803759] LustreError: 39076:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 1454.128372] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 1456.954720] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 1464.619439] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 1464.641886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:225 [ 1465.904911] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values == 14:59:25 (1476482365) [ 1467.275699] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 14:59:26 (1476482366) [ 1468.517384] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs == 14:59:28 (1476482368) [ 1468.900630] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [ 1470.047238] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API == 14:59:29 (1476482369) [ 1470.431038] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [ 1471.692934] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases == 14:59:31 (1476482371) [ 1472.183169] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [ 1473.400986] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size == 14:59:33 (1476482373) [ 1474.381986] Lustre: Failing over lustre-OST0000 [ 1474.654499] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.290327] Lustre: server umount lustre-OST0000 complete [ 1476.787055] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 1476.810240] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 1477.181425] LustreError: 40424:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 1477.541120] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 1484.036370] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 1484.049938] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 1484.064392] Lustre: Skipped 5 previous similar messages [ 1487.716767] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 1487.739267] Lustre: lustre-OST0000: deleting orphan objects from 0x0:232 to 0x0:257 [ 1491.133095] Lustre: DEBUG MARKER: sanity test_27F: @@@@@@ FAIL: wrong stripecount [ 1493.408071] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============== 14:59:52 (1476482392) [ 1494.785666] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================== 14:59:54 (1476482394) [ 1495.194186] Lustre: DEBUG MARKER: first d29 [ 1495.587799] Lustre: DEBUG MARKER: second d29 [ 1495.979173] Lustre: DEBUG MARKER: done [ 1497.128397] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ================ 14:59:56 (1476482396) [ 1498.293869] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ============= 14:59:57 (1476482397) [ 1499.500011] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ====== 14:59:59 (1476482399) [ 1500.805960] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ==================================== 15:00:00 (1476482400) [ 1501.985997] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ========= 15:00:01 (1476482401) [ 1503.159936] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links =============== 15:00:02 (1476482402) [ 1504.341365] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =========================== 15:00:03 (1476482403) [ 1505.500874] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ================= 15:00:05 (1476482405) [ 1506.682629] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ===== 15:00:06 (1476482406) [ 1513.979839] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================= 15:00:13 (1476482413) [ 1515.151432] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================= 15:00:14 (1476482414) [ 1516.321858] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================= 15:00:15 (1476482415) [ 1517.473960] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================= 15:00:17 (1476482417) [ 1518.683841] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir================= 15:00:18 (1476482418) [ 1519.973050] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir================= 15:00:19 (1476482419) [ 1521.177049] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file == 15:00:20 (1476482420) [ 1522.315009] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename == 15:00:21 (1476482421) [ 1530.792221] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory == 15:00:30 (1476482430) [ 1531.394968] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 1532.567797] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ======================= 15:00:32 (1476482432) [ 1533.731949] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ======================= 15:00:33 (1476482433) [ 1534.925153] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir =========== 15:00:34 (1476482434) [ 1536.107836] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir =========== 15:00:35 (1476482435) [ 1537.313283] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ====== 15:00:36 (1476482436) [ 1538.515232] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ====== 15:00:38 (1476482438) [ 1539.712653] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 == 15:00:39 (1476482439) [ 1540.909179] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 == 15:00:40 (1476482440) [ 1542.119664] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file ============= 15:00:41 (1476482441) [ 1543.333561] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file ============= 15:00:42 (1476482442) [ 1544.547521] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ========== 15:00:44 (1476482444) [ 1545.788047] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ========== 15:00:45 (1476482445) [ 1546.976974] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ======== 15:00:46 (1476482446) [ 1548.166882] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ======== 15:00:47 (1476482447) [ 1549.365984] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ == 15:00:48 (1476482448) [ 1550.573871] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ == 15:00:50 (1476482450) [ 1550.946467] Lustre: DEBUG MARKER: 32p_1 [ 1551.352496] Lustre: DEBUG MARKER: 32p_2 [ 1551.729702] Lustre: DEBUG MARKER: 32p_3 [ 1552.105046] Lustre: DEBUG MARKER: 32p_4 [ 1552.514347] Lustre: DEBUG MARKER: 32p_5 [ 1552.900211] Lustre: DEBUG MARKER: 32p_6 [ 1553.294869] Lustre: DEBUG MARKER: 32p_7 [ 1553.690900] Lustre: DEBUG MARKER: 32p_8 [ 1554.096280] Lustre: DEBUG MARKER: 32p_9 [ 1554.495582] Lustre: DEBUG MARKER: 32p_10 [ 1555.635757] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) == 15:00:55 (1476482455) [ 1556.829743] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) == 15:00:56 (1476482456) [ 1557.998925] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ====== 15:00:57 (1476482457) [ 1558.621800] Lustre: DEBUG MARKER: 33_1 [ 1558.994477] Lustre: DEBUG MARKER: 33_2 [ 1560.141961] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) == 15:00:59 (1476482459) [ 1561.509369] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) == 15:01:01 (1476482461) [ 1562.749238] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes == 15:01:02 (1476482462) [ 1564.333560] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir == 15:01:03 (1476482463) [ 1564.725127] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 1565.902540] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode == 15:01:05 (1476482465) [ 1566.289506] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 1567.444425] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory == 15:01:07 (1476482467) [ 1567.835921] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 1568.983884] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ============= 15:01:08 (1476482468) [ 1570.177325] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ======= 15:01:09 (1476482469) [ 1571.325205] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ================ 15:01:10 (1476482470) [ 1572.511834] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file =============================== 15:01:12 (1476482472) [ 1573.734819] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ==== 15:01:13 (1476482473) [ 1574.945533] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ========= 15:01:14 (1476482474) [ 1576.150663] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ================================= 15:01:15 (1476482475) [ 1577.451322] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 15:01:16 (1476482476) [ 1581.347541] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ======= 15:01:20 (1476482480) [ 1582.560220] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ===================== 15:01:22 (1476482482) [ 1583.764261] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ====================== 15:01:23 (1476482483) [ 1584.952937] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ============ 15:01:24 (1476482484) [ 1586.280577] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ============= 15:01:25 (1476482485) [ 1587.474538] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ====== 15:01:27 (1476482487) [ 1588.729555] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ============ 15:01:28 (1476482488) [ 1590.978033] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ======================= 15:01:30 (1476482490) [ 1715.228885] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ============ 15:03:34 (1476482614) [ 1717.415405] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory == 15:03:36 (1476482616) [ 1717.819992] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 1718.978831] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ===== 15:03:38 (1476482618) [ 1720.144152] Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create ============================= 15:03:39 (1476482619) [ 1723.359601] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ======== 15:03:42 (1476482622) [ 1725.768965] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ============================= 15:03:45 (1476482625) [ 1729.056781] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ================================ 15:03:48 (1476482628) [ 1730.233096] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ========================== 15:03:49 (1476482629) [ 1731.435614] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =================== 15:03:50 (1476482630) [ 1734.614243] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ================================= 15:03:54 (1476482634) [ 1737.815698] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat =============== 15:03:57 (1476482637) [ 1740.057614] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ================================ 15:03:59 (1476482639) [ 1742.303338] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ========================= 15:04:01 (1476482641) [ 1745.524586] Lustre: 2 MB is too small for debug buffer size, setting it to 128 MB. [ 1747.051773] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ========================== 15:04:06 (1476482646) [ 1749.417056] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ============================= 15:04:08 (1476482648) [ 1757.693239] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 == 15:04:17 (1476482657) [ 1760.894154] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored == 15:04:20 (1476482660) [ 1772.831462] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ========== 15:04:32 (1476482672) [ 1774.019835] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========== 15:04:33 (1476482673) [ 1774.417089] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 1775.555085] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate =============== 15:04:35 (1476482675) [ 1776.716732] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ======================= 15:04:36 (1476482676) [ 1778.883572] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 1779.293911] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 1779.702142] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 1780.094199] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 1780.490186] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously == 15:04:39 (1476482679) [ 1806.523698] Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY == 15:05:05 (1476482705) [ 1808.758778] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY == 15:05:08 (1476482708) [ 1810.425839] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY == 15:05:09 (1476482709) [ 1811.642112] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre========================== 15:05:11 (1476482711) [ 1812.840329] Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe =============== 15:05:12 (1476482712) [ 1813.228543] Lustre: DEBUG MARKER: SKIP: sanity test_44 skipping 2-stripe test [ 1814.394852] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ================================= 15:05:13 (1476482713) [ 1815.894459] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 1816.297056] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================== 15:05:15 (1476482715) [ 1817.694921] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=== 15:05:17 (1476482717) [ 1818.954432] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=== 15:05:18 (1476482718) [ 1820.176303] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) == 15:05:19 (1476482719) [ 1821.476255] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) == 15:05:20 (1476482720) [ 1822.679261] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) == 15:05:22 (1476482722) [ 1823.938622] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent == 15:05:23 (1476482723) [ 1825.505712] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ================= 15:05:24 (1476482724) [ 1826.731491] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ==== 15:05:26 (1476482726) [ 1828.605941] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit == 15:05:28 (1476482728) [ 2017.192019] Lustre: DEBUG MARKER: == sanity test 51ba: verify nlink for many subdirectory cleanup == 15:08:36 (1476482916) [ 2411.995287] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d [ 2412.397481] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit == 15:15:11 (1476483311) [ 3005.034499] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ======= 15:25:03 (1476483903) [ 3006.290701] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ========= 15:25:05 (1476483905) [ 3007.510324] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ====== 15:25:06 (1476483906) [ 3011.002188] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ============================ 15:25:09 (1476483909) [ 3012.402663] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ======================== 15:25:11 (1476483911) [ 3013.562352] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ======================= 15:25:12 (1476483912) [ 3015.582766] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ======================== 15:25:14 (1476483914) [ 3016.744842] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ======================== 15:25:15 (1476483915) [ 3017.922448] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe == 15:25:16 (1476483916) [ 3018.557361] Lustre: DEBUG MARKER: SKIP: sanity test_56a skipping other /usr/bin/lfs getstripe --obd test [ 3019.739478] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name =============================== 15:25:18 (1476483918) [ 3021.144228] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============================== 15:25:19 (1476483919) [ 3022.522002] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========= 15:25:21 (1476483921) [ 3023.719091] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============================== 15:25:22 (1476483922) [ 3025.022822] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============================== 15:25:23 (1476483923) [ 3026.364260] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============================== 15:25:25 (1476483925) [ 3027.813357] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============================== 15:25:26 (1476483926) [ 3029.172787] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============================== 15:25:28 (1476483928) [ 3030.492652] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================ 15:25:29 (1476483929) [ 3031.960800] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ================================= 15:25:30 (1476483930) [ 3033.350375] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ================================= 15:25:32 (1476483932) [ 3034.742820] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ============================ 15:25:33 (1476483933) [ 3036.496361] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works == 15:25:35 (1476483935) [ 3037.929398] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works == 15:25:36 (1476483936) [ 3039.500327] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works == 15:25:38 (1476483938) [ 3040.995586] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ========= 15:25:39 (1476483939) [ 3042.643429] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works == 15:25:41 (1476483941) [ 3043.407021] Lustre: DEBUG MARKER: SKIP: sanity test_56w 1 < 2 OSTs -- skipping [ 3044.676977] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support == 15:25:43 (1476483943) [ 3045.063750] Lustre: DEBUG MARKER: SKIP: sanity test_56x need 2 OST, skipping test [ 3046.220100] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released == 15:25:45 (1476483945) [ 3047.662012] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error == 15:25:46 (1476483946) [ 3049.278863] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir == 15:25:48 (1476483948) [ 3049.658986] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 3050.798253] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ==== 15:25:49 (1476483949) [ 3052.650907] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ===== 15:25:51 (1476483951) [ 3057.398558] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ================ 15:25:56 (1476483956) [ 3058.566516] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =========== 15:25:57 (1476483957) [ 3071.762228] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============ 15:26:10 (1476483970) [ 3072.403078] Lustre: DEBUG MARKER: test_60 run 25815 - from kernel mode [ 3103.031665] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 3103.031710] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 3103.032823] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc5aa24f7 to 0xedc7dcbfc65317a4 [ 3103.033553] Lustre: MGC10.2.4.47@tcp: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 3103.033555] Lustre: Skipped 2 previous similar messages [ 3103.111034] LustreError: Skipped 1 previous similar message [ 3103.120657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3105.996637] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========== 15:26:44 (1476484004) [ 3108.845537] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2102 to 0x0:2145 [ 3110.038591] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full == 15:26:48 (1476484008) [ 3111.449741] LustreError: 40451:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 3133.212754] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 15:27:12 (1476484032) [ 3134.270179] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created == 15:27:13 (1476484033) [ 3135.531232] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================== 15:27:14 (1476484034) [ 3136.745192] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 15:27:15 (1476484035) [ 3153.056445] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3153.078391] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 3153.091974] Lustre: Skipped 1 previous similar message [ 3198.100595] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ===== 15:28:16 (1476484096) [ 3201.994502] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ======= 15:28:20 (1476484100) [ 3203.264524] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 3203.667771] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ == 15:28:22 (1476484102) [ 3204.868643] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ====================== 15:28:23 (1476484103) [ 3206.081899] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 == 15:28:24 (1476484104) [ 3207.358246] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 == 15:28:26 (1476484106) [ 3208.578953] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count == 15:28:27 (1476484107) [ 3209.859664] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========================= 15:28:28 (1476484108) [ 3211.093189] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ===== 15:28:29 (1476484109) [ 3212.320235] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d ============================= 15:28:31 (1476484111) [ 3213.513603] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ====================== 15:28:32 (1476484112) [ 3214.717261] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)=== 15:28:33 (1476484113) [ 3215.964853] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping == 15:28:34 (1476484114) [ 3218.185257] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping == 15:28:36 (1476484116) [ 3219.909184] Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping == 15:28:38 (1476484118) [ 3231.777951] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=== 15:28:50 (1476484130) [ 3234.070970] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs == 15:28:52 (1476484132) [ 3234.470854] Lustre: DEBUG MARKER: SKIP: sanity test_65k too few OSTs [ 3235.671954] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ========================== 15:28:54 (1476484134) [ 3236.907611] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ================= 15:28:55 (1476484135) [ 3240.043297] Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) [ 3240.445704] Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b [ 3240.860594] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ======== 15:28:59 (1476484139) [ 3241.803704] Lustre: *** cfs_fail_loc=217, val=0*** [ 3241.812315] Lustre: Skipped 3 previous similar messages [ 3243.828636] Lustre: *** cfs_fail_loc=217, val=0*** [ 3243.837359] Lustre: Skipped 3 previous similar messages [ 3245.381350] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 3245.797365] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ====== 15:29:04 (1476484144) [ 3247.274758] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) == 15:29:05 (1476484145) [ 3248.999059] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) == 15:29:07 (1476484147) [ 3276.221007] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) == 15:29:34 (1476484174) [ 3277.375847] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) == 15:29:36 (1476484176) [ 3278.542657] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) == 15:29:37 (1476484177) [ 3279.688292] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 3280.091827] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation == 15:29:38 (1476484178) [ 3282.603708] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read == 15:29:41 (1476484181) [ 3282.687931] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x2000013a1:0xd1b3:0x0] object 0x0:7171 extent [0-1048575]: client csum e948a1, server csum e948a0 [ 3283.500411] Lustre: DEBUG MARKER: set checksum type to crc32 [ 3285.268342] Lustre: DEBUG MARKER: set checksum type to adler [ 3286.959480] Lustre: DEBUG MARKER: set checksum type to adler [ 3288.094560] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read == 15:29:46 (1476484186) [ 3288.214851] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x2000013a1:0xd1b4:0x0] object 0x0:7172 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [ 3292.295624] Lustre: DEBUG MARKER: SKIP: sanity test_77f skipping SLOW test 77f [ 3292.720549] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read == 15:29:51 (1476484191) [ 3293.185298] Lustre: *** cfs_fail_loc=21a, val=0*** [ 3293.195085] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x2000013a1:0xd1b6:0x0] object 0x0:7173 extent [0-1048575]: client csum e948a0, server csum 275347b7 [ 3294.879360] Lustre: *** cfs_fail_loc=21b, val=0*** [ 3297.640695] Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM == 15:29:56 (1476484196) [ 3297.914386] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 3299.225844] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 == 15:29:57 (1476484197) [ 3302.712720] Lustre: DEBUG MARKER: SKIP: sanity test_78 skipping SLOW test 78 [ 3303.095010] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ========================= 15:30:01 (1476484201) [ 3305.490160] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ====== 15:30:04 (1476484204) [ 3308.179257] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ================= 15:30:06 (1476484206) [ 3308.563723] Lustre: *** cfs_fail_loc=228, val=0*** [ 3310.162175] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ========= 15:30:08 (1476484208) [ 3310.533008] Lustre: *** cfs_fail_loc=228, val=0*** [ 3312.408561] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ================================= 15:30:11 (1476484211) [ 3316.105446] Lustre: DEBUG MARKER: == sanity test 99a: cvs init =========================================== 15:30:14 (1476484214) [ 3318.196803] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ========================================= 15:30:16 (1476484216) [ 3320.887472] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ======================================= 15:30:19 (1476484219) [ 3325.061653] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ============================================ 15:30:23 (1476484223) [ 3326.296068] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ========================================= 15:30:24 (1476484224) [ 3327.817091] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ========================================= 15:30:26 (1476484226) [ 3330.304160] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ============= 15:30:28 (1476484228) [ 3331.675671] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads ================== 15:30:30 (1476484230) [ 3353.927685] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =================== 15:30:52 (1476484252) [ 3354.300642] Lustre: DEBUG MARKER: SKIP: sanity test_101b skipping stride IO stride-ahead test [ 3355.453984] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead =================== 15:30:54 (1476484254) [ 3379.075582] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled == 15:31:17 (1476484277) [ 3547.218200] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) == 15:34:05 (1476484445) [ 3557.945201] Lustre: DEBUG MARKER: == sanity test 101f: check read-ahead for max_read_ahead_whole_mb == 15:34:16 (1476484456) [ 3559.344208] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ==================================== 15:34:17 (1476484457) [ 3560.596226] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ============== 15:34:19 (1476484459) [ 3560.998419] Lustre: DEBUG MARKER: SKIP: sanity test_102b skipping 2-stripe test [ 3562.165827] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ============= 15:34:20 (1476484460) [ 3562.573937] Lustre: DEBUG MARKER: SKIP: sanity test_102c skipping 2-stripe test [ 3563.730898] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============= 15:34:22 (1476484462) [ 3564.123675] Lustre: DEBUG MARKER: SKIP: sanity test_102d skipping N-stripe test [ 3565.281044] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ============= 15:34:23 (1476484463) [ 3565.683498] Lustre: DEBUG MARKER: SKIP: sanity test_102f skipping N-stripe test [ 3566.876090] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block == 15:34:25 (1476484465) [ 3567.304639] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 3567.708840] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 3568.108766] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 3568.511218] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3569.667755] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode == 15:34:28 (1476484468) [ 3570.304595] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [ 3571.456954] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ============== 15:34:30 (1476484470) [ 3572.661913] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ===== 15:34:31 (1476484471) [ 3573.067827] Lustre: DEBUG MARKER: SKIP: sanity test_102j skipping N-stripe test [ 3574.222656] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash == 15:34:32 (1476484472) [ 3575.498704] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ==================================== 15:34:34 (1476484474) [ 3576.909871] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========== 15:34:35 (1476484475) [ 3578.998725] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs == 15:34:37 (1476484477) [ 3580.723304] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission == 15:34:39 (1476484479) [ 3582.196927] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans == 15:34:40 (1476484480) [ 3583.605593] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values == 15:34:42 (1476484482) [ 3585.109925] Lustre: DEBUG MARKER: == sanity test 103a: acl test =========================================== 15:34:43 (1476484483) [ 3603.019831] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' == 15:35:01 (1476484501) [ 3612.960837] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) [ 3612.974823] Lustre: Skipped 3 previous similar messages [ 3628.288673] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 3628.288707] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3628.289837] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3628.349479] LustreError: Skipped 1 previous similar message [ 3628.358921] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 3628.380903] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc65317a4 to 0xedc7dcbfc65d19e3 [ 3628.580875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8497 to 0x0:8545 [ 3644.248424] Lustre: 40451:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476484536/real 1476484536] req@ffff8807fe6c0300 x1548202900760272/t0(0) o101->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 328/344 e 0 to 1 dl 1476484543 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 3644.289032] Lustre: 40451:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3644.303207] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 3644.324543] LustreError: 40451:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [ 3644.325332] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc65d19e3 to 0xedc7dcbfc65d1bc6 [ 3646.821517] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl == 15:35:45 (1476484545) [ 3652.952954] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8547 to 0x0:8577 [ 3654.407928] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =========================== 15:35:52 (1476484552) [ 3654.640678] Lustre: lustre-OST0000: Client 5690aa25-fbda-d8fe-e139-90d81d542800 (at 10.2.4.36@tcp) reconnecting [ 3655.076507] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880047f38c00.ost_server_uuid in FULL state after 0 sec [ 3656.416631] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================== 15:35:54 (1476484554) [ 3657.570393] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ========== 15:35:56 (1476484556) [ 3658.802710] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ========== 15:35:57 (1476484557) [ 3659.979363] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========== 15:35:58 (1476484558) [ 3661.228526] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ========== 15:35:59 (1476484559) [ 3669.336709] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 3669.351934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3669.377391] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3672.463224] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========= 15:36:11 (1476484571) [ 3673.681346] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir == 15:36:12 (1476484572) [ 3674.886832] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG == 15:36:13 (1476484573) [ 3677.140722] Lustre: DEBUG MARKER: == sanity test 110: filename length checking == 15:36:15 (1476484575) [ 3678.411984] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 3678.811495] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ===================== 15:36:17 (1476484577) [ 3679.221700] Lustre: DEBUG MARKER: SKIP: sanity test_116a 1 < 2 OSTs [ 3680.375468] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass == 15:36:18 (1476484578) [ 3682.796797] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ============ 15:36:21 (1476484581) [ 3684.071705] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============ 15:36:22 (1476484582) [ 3685.649495] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============ 15:36:24 (1476484584) [ 3686.174288] Lustre: *** cfs_fail_loc=217, val=0*** [ 3688.104763] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============ 15:36:26 (1476484586) [ 3688.630905] LustreError: 51795:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 sleeping for 1000ms [ 3689.646620] LustreError: 51795:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 awake [ 3692.238466] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ============ 15:36:30 (1476484590) [ 3692.693497] LustreError: 51795:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 25000ms [ 3717.720362] LustreError: 51795:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 3719.563016] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ============ 15:36:58 (1476484618) [ 3721.181538] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ============ 15:36:59 (1476484619) [ 3722.783793] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ============ 15:37:01 (1476484621) [ 3723.228396] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3726.246219] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3726.254920] Lustre: Skipped 1 previous similar message [ 3733.267587] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3733.276217] Lustre: Skipped 1 previous similar message [ 3734.908541] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ============ 15:37:13 (1476484633) [ 3743.013233] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ============ 15:37:21 (1476484641) [ 3743.554277] Lustre: *** cfs_fail_loc=220, val=0*** [ 3743.562805] Lustre: Skipped 3 previous similar messages [ 3745.293078] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =========== 15:37:23 (1476484643) [ 3755.494458] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========== 15:37:33 (1476484653) [ 3757.083106] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =========== 15:37:35 (1476484655) [ 3758.432348] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount == 15:37:36 (1476484656) [ 3759.957302] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount == 15:37:38 (1476484658) [ 3760.352594] Lustre: DEBUG MARKER: SKIP: sanity test_119b skipping 2-stripe test [ 3761.546517] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole == 15:37:40 (1476484660) [ 3763.066702] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed == 15:37:41 (1476484661) [ 3764.753362] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 3767.916831] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test == 15:37:46 (1476484666) [ 3769.626872] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test == 15:37:48 (1476484668) [ 3771.323349] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test == 15:37:49 (1476484669) [ 3772.949928] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test == 15:37:51 (1476484671) [ 3774.642575] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test == 15:37:53 (1476484673) [ 3781.399096] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test == 15:37:59 (1476484679) [ 3788.280839] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test == 15:38:06 (1476484686) [ 3802.799399] Lustre: 43115:0:(osd_handler.c:1708:osd_trans_start()) lustre-OST0000: credits 3203 > trans_max 2624 [ 3802.813367] Lustre: 43115:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 128/1024/0, destroy: 0/0/0 [ 3802.826619] Lustre: 43115:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 3802.839591] Lustre: 43115:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 2/2/0 [ 3802.852992] Lustre: 43115:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 3802.865664] Lustre: 43115:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3802.877958] Pid: 43115, comm: ll_ost01_010 [ 3802.884113] Call Trace: [ 3802.891545] [] libcfs_debug_dumpstack+0x53/0x80 [libcfs] [ 3802.900814] [] osd_trans_start+0x40e/0x440 [osd_ldiskfs] [ 3802.909984] [] ofd_precreate_objects+0x943/0x18f0 [ofd] [ 3802.919034] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 3802.927699] [] ofd_create_hdl+0x43e/0x28f0 [ofd] [ 3802.936020] [] ? lustre_pack_reply_v2+0x183/0x280 [ptlrpc] [ 3802.945211] [] tgt_request_handle+0x915/0x1320 [ptlrpc] [ 3802.954088] [] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [ 3802.963781] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 3802.972494] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 3802.980994] [] ptlrpc_main+0xaa0/0x1de0 [ptlrpc] [ 3802.989108] [] ? ptlrpc_main+0x0/0x1de0 [ptlrpc] [ 3802.997186] [] kthread+0xcf/0xe0 [ 3803.003689] [] ? kthread+0x0/0xe0 [ 3803.010266] [] ret_from_fork+0x58/0x90 [ 3803.017298] [] ? kthread+0x0/0xe0 [ 3819.217797] Lustre: DEBUG MARKER: == sanity test 121: read cancel race =========== 15:38:37 (1476484717) [ 3820.402612] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work == 15:38:38 (1476484718) [ 3820.794343] Lustre: DEBUG MARKER: testing on UP system. Performance may be not as good as expected. [ 3821.530271] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [ 3822.027628] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [ 3825.043042] Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec [ 3826.145550] Lustre: DEBUG MARKER: ls 1000 files with statahead: 0 sec [ 3862.549726] Lustre: DEBUG MARKER: ls 10000 files without statahead: 24 sec [ 3869.897183] Lustre: DEBUG MARKER: ls 10000 files with statahead: 6 sec [ 3870.294337] Lustre: DEBUG MARKER: ls done [ 3879.288456] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 8 seconds [ 3879.670274] Lustre: DEBUG MARKER: rm done [ 3880.904922] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) == 15:39:39 (1476484779) [ 3885.854399] Lustre: DEBUG MARKER: ls done [ 3888.130187] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ========================================= 15:39:46 (1476484786) [ 3888.448643] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 3893.504122] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880047f38c00 [ 3893.887254] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880047f38c00 [ 3894.258272] Lustre: DEBUG MARKER: LRU=2003 [ 3894.629025] Lustre: DEBUG MARKER: LIMIT=1600949 [ 3895.007598] Lustre: DEBUG MARKER: LVF=959130 [ 3895.404853] Lustre: DEBUG MARKER: OLD_LVF=1 [ 3895.778982] Lustre: DEBUG MARKER: Sleep 50 sec [ 3946.240835] Lustre: DEBUG MARKER: Dropped 777 locks in 50s [ 3946.632318] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 3950.302159] Lustre: DEBUG MARKER: SKIP: sanity test_124b skipping SLOW test 124b [ 3950.701255] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks == 15:40:49 (1476484849) [ 3972.679890] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs == 15:41:11 (1476484871) [ 3974.066473] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account == 15:41:12 (1476484872) [ 3975.254944] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane == 15:41:13 (1476484873) [ 3976.971337] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane == 15:41:15 (1476484875) [ 3978.229862] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's == 15:41:16 (1476484876) [ 3979.421673] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ========================== 15:41:17 (1476484877) [ 4057.109095] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) == 15:42:35 (1476484955) [ 4058.384700] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) == 15:42:36 (1476484956) [ 4058.764824] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping FIEMAP on 2-stripe file test [ 4059.923112] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) == 15:42:38 (1476484958) [ 4060.296303] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping FIEMAP on 2-stripe file [ 4061.476376] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) == 15:42:39 (1476484959) [ 4061.883457] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [ 4063.073988] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) == 15:42:41 (1476484961) [ 4063.453705] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping continuation FIEMAP test [ 4064.647231] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv == 15:42:42 (1476484962) [ 4066.015558] Lustre: DEBUG MARKER: == sanity test 131b: test append writev == 15:42:44 (1476484964) [ 4067.390100] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects == 15:42:45 (1476484965) [ 4068.582792] Lustre: DEBUG MARKER: == sanity test 131d: test short read == 15:42:46 (1476484966) [ 4069.783997] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole == 15:42:48 (1476484968) [ 4070.982290] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ========================================== 15:42:49 (1476484969) [ 4075.484630] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ==================================== 15:42:53 (1476484973) [ 4078.500840] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ========================================== 15:42:56 (1476484976) [ 4095.908198] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================== 15:43:14 (1476484994) [ 4100.818533] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =================== 15:43:19 (1476484999) [ 4103.852798] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 15:43:22 (1476485002) [ 4108.814271] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc == 15:43:27 (1476485007) [ 4111.588313] LustreError: 94233:0:(ldlm_resource.c:109:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [ 4111.644202] LustreError: 94265:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Can't parse user input, rc = -14 [ 4111.672096] LustreError: 94275:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) invalid value written [ 4111.828854] LustreError: 94355:0:(genops.c:1519:obd_export_evict_by_uuid()) lustre-OST0000: can't disconnect : no exports found [ 4111.979778] LustreError: 94484:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) srv-lustre-OST0000: invalid FID sequence width: rc = -14 [ 4112.006760] LustreError: 94490:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) cannot add nodemap config to non-existing MGS. [ 4112.032608] LustreError: 94498:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [ 4112.063341] Lustre: 94515:0:(libcfs_string.c:131:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 4112.083236] Lustre: 0 MB is too small for debug buffer size, setting it to 128 MB. [ 4119.556091] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 4119.571188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4132.307373] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) [ 4132.321367] Lustre: Skipped 6 previous similar messages [ 4134.748135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4144.568359] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 4144.582346] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 4144.604988] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc65d1bc6 to 0xedc7dcbfc67d3541 [ 4151.421069] Lustre: lustre-OST0000: deleting orphan objects from 0x0:33274 to 0x0:34017 [ 4152.583999] LustreError: 40451:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 4152.596962] LustreError: 40451:0:(mgc_request.c:571:do_requeue()) Skipped 1 previous similar message [ 4174.673949] Lustre: 94670:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476485067/real 1476485067] req@ffff8803f51e0000 x1548202900927616/t0(0) o251->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476485073 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4175.146776] Lustre: server umount lustre-OST0000 complete [ 4197.582911] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 4197.619276] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 4198.011028] LustreError: 94874:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 4200.829155] Lustre: lustre-OST0000: deleting orphan objects from 0x0:33274 to 0x0:34049 [ 4228.160026] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 4234.572495] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold == 15:45:32 (1476485132) [ 4251.202265] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb == 15:45:49 (1476485149) [ 4359.205715] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ====== 15:47:37 (1476485257) [ 4361.102327] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests == 15:47:39 (1476485259) [ 4361.649173] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 4361.664128] Lustre: Skipped 4 previous similar messages [ 4364.180676] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ================================= 15:47:42 (1476485262) [ 4370.448214] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ============================== 15:47:48 (1476485268) [ 4371.683323] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ========================= 15:47:49 (1476485269) [ 4373.170388] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks == 15:47:51 (1476485271) [ 4374.636032] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID == 15:47:52 (1476485272) [ 4377.175034] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory == 15:47:55 (1476485275) [ 4377.796947] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 4378.962403] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments == 15:47:57 (1476485277) [ 4380.773737] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid == 15:47:58 (1476485278) [ 4382.463023] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 15:48:00 (1476485280) [ 4383.881284] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 15:48:02 (1476485282) [ 4385.228613] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests == 15:48:03 (1476485283) [ 4444.983039] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on == 15:49:03 (1476485343) [ 4447.268045] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off == 15:49:05 (1476485345) [ 4449.294936] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on == 15:49:07 (1476485347) [ 4451.388940] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off == 15:49:09 (1476485349) [ 4453.575289] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on == 15:49:11 (1476485351) [ 4683.282131] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off == 15:53:01 (1476485581) [ 4911.411682] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on == 15:56:49 (1476485809) [ 5145.294821] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off == 16:00:43 (1476486043) [ 5382.578527] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables == 16:04:40 (1476486280) [ 5384.276470] Lustre: DEBUG MARKER: Turn on read and write cache [ 5385.386547] Lustre: DEBUG MARKER: Write data and read it back. [ 5385.751833] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 5386.888694] Lustre: DEBUG MARKER: cache hits:: before: 655392, after: 655395 [ 5387.277642] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 5388.018148] Lustre: DEBUG MARKER: cache hits:: before: 655395, after: 655398 [ 5388.404531] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 5389.512548] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 5390.642662] Lustre: DEBUG MARKER: cache hits:: before: 655398, after: 655401 [ 5391.014471] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 5391.804180] Lustre: DEBUG MARKER: cache hits:: before: 655401, after: 655401 [ 5392.187235] Lustre: DEBUG MARKER: Write data and read it back. [ 5392.567652] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 5393.751729] Lustre: DEBUG MARKER: cache hits:: before: 655401, after: 655404 [ 5394.153355] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 5394.975996] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [ 5395.353520] Lustre: DEBUG MARKER: Turn off read and write cache [ 5396.473275] Lustre: DEBUG MARKER: Write data and read it back [ 5396.859189] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 5398.089490] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [ 5398.482040] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 5399.594505] Lustre: DEBUG MARKER: Write data and read it back [ 5399.970247] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 5401.230646] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [ 5401.593226] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 5402.726981] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655407 [ 5403.890008] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity == 16:05:01 (1476486301) [ 5417.720025] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 5423.723352] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 5423.723384] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 5423.724269] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc67d36c9 to 0xedc7dcbfc68648df [ 5423.776260] LustreError: Skipped 1 previous similar message [ 5423.785120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5423.809423] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5426.832705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35691 to 0x0:35745 [ 5428.238029] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog == 16:05:25 (1476486325) [ 5428.945101] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 5430.561539] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event == 16:05:28 (1476486328) [ 5433.490199] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event == 16:05:31 (1476486331) [ 5433.897399] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 5435.096536] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity == 16:05:32 (1476486332) [ 5440.011930] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory == 16:05:37 (1476486337) [ 5440.391749] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 5441.544473] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags == 16:05:39 (1476486339) [ 5443.664729] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity == 16:05:41 (1476486341) [ 5445.036365] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity == 16:05:42 (1476486342) [ 5445.438068] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 5446.595868] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep == 16:05:44 (1476486344) [ 5451.187453] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock == 16:05:48 (1476486348) [ 5451.569801] Lustre: DEBUG MARKER: creating a 10 Mb file [ 5452.770375] Lustre: DEBUG MARKER: starting reads [ 5453.164819] Lustre: DEBUG MARKER: truncating the file [ 5453.707614] Lustre: DEBUG MARKER: killing dd [ 5454.097916] Lustre: DEBUG MARKER: removing the temporary file [ 5455.251642] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ======================= 16:05:52 (1476486352) [ 5458.450614] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ======== 16:05:56 (1476486356) [ 5462.685970] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc == 16:06:00 (1476486360) [ 5462.759098] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 5462.759100] Lustre: Skipped 2 previous similar messages [ 5465.512270] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 16:06:03 (1476486363) [ 5465.858647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.263634] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT == 16:06:08 (1476486368) [ 5472.065585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5485.172607] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ========================== 16:06:22 (1476486382) [ 5501.134918] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ================== 16:06:38 (1476486398) [ 5511.180658] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========== 16:06:48 (1476486408) [ 5513.150430] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap == 16:06:50 (1476486410) [ 5514.578063] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) == 16:06:52 (1476486412) [ 5516.453329] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap == 16:06:54 (1476486414) [ 5537.488580] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap == 16:07:15 (1476486435) [ 5538.746622] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps == 16:07:16 (1476486436) [ 5540.209682] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support == 16:07:17 (1476486437) [ 5542.714402] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change == 16:07:20 (1476486440) [ 5546.471254] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file == 16:07:24 (1476486444) [ 5548.376445] Lustre: DEBUG MARKER: == sanity test 200: OST pools == 16:07:26 (1476486446) [ 5583.266870] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes =================== 16:08:00 (1476486480) [ 5584.493033] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ============= 16:08:02 (1476486482) [ 5585.670806] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ============= 16:08:03 (1476486483) [ 5586.856787] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size =============== 16:08:04 (1476486484) [ 5588.109238] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes =================== 16:08:05 (1476486485) [ 5589.324621] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ============= 16:08:06 (1476486486) [ 5590.561079] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset ============= 16:08:08 (1476486488) [ 5591.749450] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size =============== 16:08:09 (1476486489) [ 5592.907799] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats == 16:08:10 (1476486490) [ 5601.485832] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [ 5601.855820] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.20770 [ 5602.518089] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [ 5602.905317] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.4166 [ 5603.532514] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [ 5603.915812] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.599 [ 5604.591008] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [ 5604.983895] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.19284 [ 5605.624596] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [ 5606.026051] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.13255 [ 5606.686884] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [ 5607.056925] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.21698 [ 5608.021926] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [ 5608.420813] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.4503 [ 5609.606166] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [ 5609.994355] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.2620 [ 5610.806548] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [ 5611.201706] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.10363 [ 5612.168864] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [ 5612.552112] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.1642 [ 5613.238336] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [ 5613.617933] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.17858 [ 5632.248551] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug == 16:08:49 (1476486529) [ 5633.419434] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse == 16:08:51 (1476486531) [ 5634.656103] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open == 16:08:52 (1476486532) [ 5635.959946] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open == 16:08:53 (1476486533) [ 5655.835941] Lustre: 23245:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476486546/real 1476486546] req@ffff88041a8db900 x1548202900933120/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476486553 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5655.875552] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5659.450312] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 5661.901877] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476486553/real 1476486553] req@ffff88041a8db000 x1548202900933152/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476486559 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5662.902394] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 5662.923858] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc68648df to 0xedc7dcbfc69a3180 [ 5671.954489] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 5684.183873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:49793 to 0x0:50049 [ 5685.234721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 23 sec [ 5712.947571] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 5712.947789] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5712.982332] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 5713.003840] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc69a3180 to 0xedc7dcbfc69a33d3 [ 5719.001523] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 5730.502551] Lustre: lustre-OST0000: deleting orphan objects from 0x0:49793 to 0x0:50081 [ 5731.435517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 21 sec [ 5733.798702] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly == 16:10:31 (1476486631) [ 5741.792005] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ============================================== 16:10:39 (1476486639) [ 5743.443375] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 == 16:10:40 (1476486640) [ 5745.039367] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 == 16:10:42 (1476486642) [ 5748.770195] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 == 16:10:46 (1476486646) [ 5749.985003] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly == 16:10:47 (1476486647) [ 5754.084818] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') == 16:10:51 (1476486651) [ 5756.190943] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================= 16:10:53 (1476486653) [ 5756.588049] Lustre: DEBUG MARKER: creating a 10 Mb file [ 5757.700086] Lustre: DEBUG MARKER: starting reads [ 5758.265679] Lustre: DEBUG MARKER: truncating the file [ 5758.659272] Lustre: DEBUG MARKER: killing dd [ 5759.044522] Lustre: DEBUG MARKER: removing the temporary file [ 5760.201699] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND == 16:10:57 (1476486657) [ 5761.901124] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST == 16:10:59 (1476486659) [ 5765.537721] Lustre: *** cfs_fail_loc=229, val=0*** [ 5770.549299] Lustre: *** cfs_fail_loc=229, val=0*** [ 5775.561609] Lustre: *** cfs_fail_loc=229, val=0*** [ 5775.570231] Lustre: Skipped 1 previous similar message [ 5780.582048] Lustre: *** cfs_fail_loc=229, val=0*** [ 5785.401681] Lustre: *** cfs_fail_loc=229, val=0*** [ 5785.410063] Lustre: Skipped 1 previous similar message [ 5797.844319] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM == 16:11:35 (1476486695) [ 5824.977341] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================== 16:12:02 (1476486722) [ 5826.257098] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure =============== 16:12:03 (1476486723) [ 5827.548908] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ========================= 16:12:05 (1476486725) [ 5828.798632] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure == 16:12:06 (1476486726) [ 5828.839870] Lustre: lustre-OST0000: Client 9566855b-9291-df57-6d58-e1856d82319f (at 10.2.4.36@tcp) reconnecting [ 5828.854287] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 5828.868861] Lustre: Skipped 4 previous similar messages [ 5830.393486] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure == 16:12:07 (1476486727) [ 5830.498984] Lustre: lustre-OST0000: Client 9566855b-9291-df57-6d58-e1856d82319f (at 10.2.4.36@tcp) reconnecting [ 5831.495163] LustreError: 97213:0:(ldlm_lib.c:3238:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8800aacebc50 x1548206827156416/t0(0) o3->9566855b-9291-df57-6d58-e1856d82319f@10.2.4.36@tcp:-1/-1 lens 608/432 e 0 to 0 dl 1476486771 ref 1 fl Interpret:/0/0 rc 0/0 [ 5831.530702] Lustre: lustre-OST0000: Bulk IO read error with 9566855b-9291-df57-6d58-e1856d82319f (at 10.2.4.36@tcp), client will retry: rc -110 [ 5832.697205] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC == 16:12:10 (1476486730) [ 5845.624617] Lustre: lustre-OST0000: haven't heard from client 0e3d8e33-6964-46a6-8c50-c910a4248a28 (at 10.2.4.37@tcp) in 15 seconds. I think it's dead, and I am evicting it. exp ffff8803f1c22400, cur 1476486743 expire 1476486737 last 1476486728 [ 5863.020420] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 5863.020438] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5863.021280] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5863.080225] LustreError: Skipped 1 previous similar message [ 5863.089452] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 5863.110714] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc69a33d3 to 0xedc7dcbfc69ae5ea [ 5864.379508] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe == 16:12:41 (1476486761) [ 5869.245456] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 6440.945682] Lustre: DEBUG MARKER: sanity test_225a: @@@@@@ FAIL: mds-survey with zero-stripe failed [ 6446.240838] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 == 16:22:23 (1476487343) [ 6458.701556] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type == 16:22:35 (1476487355) [ 6460.037150] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir == 16:22:37 (1476487357) [ 6460.400546] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 6461.566416] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM == 16:22:38 (1476487358) [ 6462.786860] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks == 16:22:39 (1476487359) [ 6517.226742] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart == 16:23:34 (1476487414) [ 6569.745280] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 6569.758272] Lustre: Skipped 5 previous similar messages [ 6570.363565] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 6570.363592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6570.403111] LustreError: Skipped 1 previous similar message [ 6570.412789] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 6570.435055] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc69ae5ea to 0xedc7dcbfc6b59f44 [ 6579.328148] Lustre: lustre-OST0000: deleting orphan objects from 0x0:73502 to 0x0:74561 [ 6579.734997] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 6589.794704] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf == 16:24:46 (1476487486) [ 6595.376081] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6658.725943] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files == 16:25:55 (1476487555) [ 6659.358062] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [ 6660.567049] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory == 16:25:57 (1476487557) [ 6660.958131] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 6662.093325] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory == 16:25:59 (1476487559) [ 6662.483325] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 6663.641563] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed == 16:26:00 (1476487560) [ 6664.045026] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 6665.171303] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory == 16:26:02 (1476487562) [ 6665.559065] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [ 6666.762045] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files == 16:26:03 (1476487563) [ 6667.157730] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 6668.299106] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 16:26:05 (1476487565) [ 6668.705940] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 6669.907318] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT == 16:26:06 (1476487566) [ 6670.308543] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 6671.663382] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root == 16:26:08 (1476487568) [ 6672.048300] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 6673.565190] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes == 16:26:10 (1476487570) [ 6673.964925] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 6675.163814] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC == 16:26:12 (1476487572) [ 6678.566489] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer == 16:26:15 (1476487575) [ 6683.249751] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount == 16:26:20 (1476487580) [ 6684.781181] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds == 16:26:21 (1476487581) [ 6686.630218] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds == 16:26:23 (1476487583) [ 6688.023469] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM == 16:26:25 (1476487585) [ 6688.591578] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.44 too old [ 6689.759050] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly == 16:26:26 (1476487586) [ 6693.256527] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file == 16:26:30 (1476487590) [ 6694.622158] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls == 16:26:31 (1476487591) [ 6695.863830] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency == 16:26:32 (1476487592) [ 6697.433571] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test == 16:26:34 (1476487594) [ 6720.982534] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly == 16:26:58 (1476487618) [ 6731.884160] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly == 16:27:08 (1476487628) [ 6736.454514] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) == 16:27:13 (1476487633) [ 6736.853820] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 6738.128651] Lustre: DEBUG MARKER: == sanity test 241: bio vs dio == 16:27:15 (1476487635) [ 6748.637926] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio == 16:27:25 (1476487645) [ 6754.770999] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable == 16:27:31 (1476487651) [ 6756.429374] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests == 16:27:33 (1476487653) [ 6770.931360] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests == 16:27:47 (1476487667) [ 6786.877857] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs == 16:28:03 (1476487683) [ 6788.032772] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length == 16:28:05 (1476487685) [ 6789.185019] Lustre: *** cfs_fail_loc=234, val=0*** [ 6789.193223] Lustre: Skipped 23 previous similar messages [ 6790.337230] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size == 16:28:07 (1476487687) [ 6791.502791] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit == 16:28:08 (1476487688) [ 6792.783854] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly == 16:28:09 (1476487689) [ 6794.018371] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool == 16:28:11 (1476487691) [ 6796.240775] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state == 16:28:13 (1476487693) [ 6820.484980] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 6820.485009] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6820.485916] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6820.544396] LustreError: Skipped 1 previous similar message [ 6820.553637] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 6820.575464] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc6b59f44 to 0xedc7dcbfc6dcb776 [ 6821.328462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103602 to 0x0:103713 [ 6823.260336] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test == 16:28:40 (1476487720) [ 6823.645162] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 6824.825418] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir == 16:28:41 (1476487721) [ 6825.216248] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 6826.365069] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 6826.743776] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 6827.900621] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory == 16:28:44 (1476487724) [ 6828.300127] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 6828.902322] LustreError: 94901:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [ 6829.451796] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory == 16:28:46 (1476487726) [ 6830.083289] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 6831.224341] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory == 16:28:48 (1476487728) [ 6831.852239] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 6833.025252] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory == 16:28:50 (1476487730) [ 6833.651361] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 6834.796090] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory == 16:28:51 (1476487731) [ 6835.418602] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 6836.561820] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory == 16:28:53 (1476487733) [ 6837.185602] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 6838.318785] Lustre: DEBUG MARKER: == sanity test 300j: test large update record == 16:28:55 (1476487735) [ 6838.940659] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 6840.079261] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory == 16:28:57 (1476487737) [ 6840.692893] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 6841.821211] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout == 16:28:58 (1476487738) [ 6842.477881] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 6843.654176] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS == 16:29:00 (1476487740) [ 6845.370097] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA == 16:29:02 (1476487742) [ 6846.022342] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 6847.210132] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 6847.607141] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space == 16:29:04 (1476487744) [ 6847.984261] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 6849.135474] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory == 16:29:06 (1476487746) [ 6849.533759] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 6850.861739] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file == 16:29:07 (1476487747) [ 6851.238828] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 6852.369931] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open == 16:29:09 (1476487749) [ 6852.778871] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 6853.920379] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links == 16:29:10 (1476487750) [ 6854.339352] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 6855.507367] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link == 16:29:12 (1476487752) [ 6858.164427] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled == 16:29:15 (1476487755) [ 6859.364720] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively == 16:29:16 (1476487756) [ 6860.657669] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error == 16:29:17 (1476487757) [ 6861.842370] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea == 16:29:18 (1476487758) [ 6863.326664] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread == 16:29:20 (1476487760) [ 6870.509135] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 6870.524098] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6877.511541] Lustre: 23240:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476487767/real 1476487767] req@ffff88041a8d8000 x1548202900943440/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476487774 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6877.551736] Lustre: 23240:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6877.565714] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 6879.257712] Lustre: server umount lustre-OST0000 complete [ 6902.799417] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 6902.827024] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 6903.238455] LustreError: 121332:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [ 6907.281207] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103724 to 0x0:103745 [ 6933.446996] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 6939.870612] Lustre: DEBUG MARKER: == sanity test complete, duration 6156 sec == 16:30:36 (1476487836) [ 7285.529351] Lustre: 23238:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488175/real 1476488175] req@ffff88041a8dda00 x1548202901061680/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476488182 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7285.570848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7285.595597] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 7291.619273] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488182/real 1476488182] req@ffff88041a8de000 x1548202901061696/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476488188 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7291.661290] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7321.633713] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488207/real 1476488207] req@ffff88041a8de600 x1548202901061728/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488218 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7321.674920] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7351.648176] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488232/real 1476488232] req@ffff88041a8dec00 x1548202901061760/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488248 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7351.689488] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7381.662649] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488257/real 1476488257] req@ffff88041a8df500 x1548202901061792/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488278 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7381.704025] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7411.677130] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488282/real 1476488282] req@ffff88041a8df800 x1548202901061824/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488308 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7411.718485] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7466.703682] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488332/real 1476488332] req@ffff88041a8dfb00 x1548202901061856/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488363 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7466.744967] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7576.756827] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488432/real 1476488432] req@ffff88041a8def00 x1548202901061920/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7576.798024] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7765.848278] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488607/real 1476488607] req@ffff88041a8ddd00 x1548202901062016/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488662 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7765.889361] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 8065.993706] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476488907/real 1476488907] req@ffff88041a8dc800 x1548202901062144/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476488962 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8066.034888] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 8591.248842] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476489432/real 1476489432] req@ffff88041a8d9b00 x1548202901062368/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476489487 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8591.290042] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 8897.937937] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 8897.950951] Lustre: Skipped 9 previous similar messages [ 8936.417881] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 8936.440611] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0xedc7dcbfc6dcbd7a to 0x20295361fde8dd6d [ 8939.660438] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103724 to 0x0:103777 [ 8979.741462] Lustre: DEBUG MARKER: auster : @@@@@@ FAIL: server onyx-25 environments are insane! [ 9042.337130] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 17:05:38 PDT 2016 [ 9045.961936] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 9048.566360] Lustre: DEBUG MARKER: sanity : @@@@@@ FAIL: unable to write to /mnt/lustre/d0_runas_test as UID 500. [ 9174.045289] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 17:07:49 PDT 2016 [ 9177.811398] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 9180.297803] Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b [ 9180.676114] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o [ 9181.182340] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 17:07:57 (1476490077) [ 9182.356290] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 17:07:58 (1476490078) [ 9183.510430] Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 17:07:59 (1476490079) [ 9184.659922] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 17:08:00 (1476490080) [ 9185.876831] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ====================== 17:08:01 (1476490081) [ 9187.051698] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ======================= 17:08:02 (1476490082) [ 9188.239663] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) == 17:08:04 (1476490084) [ 9189.422821] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============== 17:08:05 (1476490085) [ 9190.733544] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ==== 17:08:06 (1476490086) [ 9191.942088] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ==== 17:08:07 (1476490087) [ 9193.137834] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ==== 17:08:08 (1476490088) [ 9194.334835] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? == 17:08:10 (1476490090) [ 9195.646655] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) == 17:08:11 (1476490091) [ 9196.865449] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ====== 17:08:12 (1476490092) [ 9198.016261] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ======= 17:08:13 (1476490093) [ 9199.207993] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ========= 17:08:15 (1476490095) [ 9200.378933] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================== 17:08:16 (1476490096) [ 9201.553216] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ======== 17:08:17 (1476490097) [ 9202.726243] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============== 17:08:18 (1476490098) [ 9204.132307] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ========== 17:08:19 (1476490099) [ 9205.319947] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ========== 17:08:21 (1476490101) [ 9206.511767] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ========= 17:08:22 (1476490102) [ 9207.694742] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ============ 17:08:23 (1476490103) [ 9208.886921] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f =================== 17:08:24 (1476490104) [ 9210.045981] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ==================== 17:08:25 (1476490105) [ 9211.221756] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ================ 17:08:27 (1476490107) [ 9212.432412] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ====== 17:08:28 (1476490108) [ 9213.594007] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ========================== 17:08:29 (1476490109) [ 9214.764273] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ====== 17:08:30 (1476490110) [ 9215.925587] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================== 17:08:31 (1476490111) [ 9217.140703] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries == 17:08:32 (1476490112) [ 9219.561851] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug == 17:08:35 (1476490115) [ 9221.080662] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink == 17:08:36 (1476490116) [ 9222.614270] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled =========================== 17:08:38 (1476490118) [ 9223.880378] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========== 17:08:39 (1476490119) [ 9225.108888] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink == 17:08:40 (1476490120) [ 9251.133624] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [ 9251.146871] Lustre: Skipped 2 previous similar messages [ 9261.576248] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 9261.576278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9261.601811] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9261.639779] LustreError: Skipped 1 previous similar message [ 9261.649800] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 9261.672502] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361fde8dd6d to 0x20295361fdeaaf41 [ 9270.306534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103801 to 0x0:103841 [ 9271.424076] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir == 17:09:27 (1476490167) [ 9272.501895] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [ 9273.654363] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 17:09:29 (1476490169) [ 9293.590726] Lustre: 23248:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476490182/real 1476490182] req@ffff88041a8dd400 x1548202901063632/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476490189 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9293.633450] Lustre: 23248:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 9293.648151] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9314.855486] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103843 to 0x0:103873 [ 9315.373301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 18 sec [ 9318.445645] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ================================ 17:10:14 (1476490214) [ 9318.685749] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [ 9318.685866] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9318.722912] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [ 9318.745785] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361fdeaaf41 to 0x20295361fdeab27b [ 9319.695443] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ============= 17:10:15 (1476490215) [ 9320.901755] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ================ 17:10:16 (1476490216) [ 9322.035723] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ==== 17:10:17 (1476490217) [ 9323.195324] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================ 17:10:18 (1476490218) [ 9324.306120] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... ============================= 17:10:20 (1476490220) [ 9324.699305] Lustre: DEBUG MARKER: 1 done [ 9325.088742] Lustre: DEBUG MARKER: 2 done [ 9325.455814] Lustre: DEBUG MARKER: 3 done [ 9326.577241] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================== 17:10:22 (1476490222) [ 9327.807944] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user ================= 17:10:23 (1476490223) [ 9329.226494] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============================ 17:10:25 (1476490225) [ 9330.376929] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ============================ 17:10:26 (1476490226) [ 9331.599631] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target == 17:10:27 (1476490227) [ 9332.779531] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target == 17:10:28 (1476490228) [ 9333.916805] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target == 17:10:29 (1476490229) [ 9335.107433] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 17:10:30 (1476490230) [ 9336.287540] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ======== 17:10:32 (1476490232) [ 9337.458581] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ====== 17:10:33 (1476490233) [ 9338.630007] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ======== 17:10:34 (1476490234) [ 9339.802723] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e == 17:10:35 (1476490235) [ 9340.968295] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a == 17:10:36 (1476490236) [ 9342.141748] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================== 17:10:37 (1476490237) [ 9343.323170] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ========= 17:10:39 (1476490239) [ 9344.516277] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ========================== 17:10:40 (1476490240) [ 9345.662539] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =========== 17:10:41 (1476490241) [ 9346.810925] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) == 17:10:42 (1476490242) [ 9347.998791] Lustre: DEBUG MARKER: SKIP: sanity test_24o skipping SLOW test 24o [ 9348.398096] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b == 17:10:44 (1476490244) [ 9349.579354] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ===== 17:10:45 (1476490245) [ 9350.757743] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ======= 17:10:46 (1476490246) [ 9351.941373] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c === 17:10:47 (1476490247) [ 9353.102425] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a === 17:10:48 (1476490248) [ 9354.254111] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 17:10:50 (1476490250) [ 9355.525394] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) == 17:10:51 (1476490251) [ 9718.795274] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb == 17:16:54 (1476490614) [ 9720.014020] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link == 17:16:55 (1476490615) [ 9720.386542] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 9721.550564] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed == 17:16:57 (1476490617) [ 9721.942915] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 9723.095583] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. == 17:16:58 (1476490618) [ 9738.268108] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries == 17:17:13 (1476490633) [ 9738.659780] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 9739.792109] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir == 17:17:15 (1476490635) [ 9740.198703] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 9741.344212] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [ 9741.722348] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link == 17:17:17 (1476490637) [ 9742.106269] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 9743.226635] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ================= 17:17:18 (1476490638) [ 9744.412178] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ================= 17:17:20 (1476490640) [ 9745.569224] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ========================= 17:17:21 (1476490641) [ 9746.739274] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ======== 17:17:22 (1476490642) [ 9747.894502] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================== 17:17:23 (1476490643) [ 9749.062735] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ======== 17:17:24 (1476490644) [ 9750.205388] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ======== 17:17:25 (1476490645) [ 9751.376977] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink === 17:17:26 (1476490646) [ 9752.578482] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ==================================== 17:17:28 (1476490648) [ 9753.001878] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [ 9754.154111] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 17:17:29 (1476490649) [ 9754.542291] Lustre: DEBUG MARKER: SKIP: sanity test_27b skipping 2-stripe test [ 9755.657037] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================== 17:17:31 (1476490651) [ 9756.854539] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ======== 17:17:32 (1476490652) [ 9758.335446] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) == 17:17:33 (1476490653) [ 9759.572101] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects == 17:17:35 (1476490655) [ 9760.717157] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects == 17:17:36 (1476490656) [ 9761.874182] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) == 17:17:37 (1476490657) [ 9763.055716] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps =============== 17:17:38 (1476490658) [ 9764.218136] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) == 17:17:39 (1476490659) [ 9765.358487] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 9765.737623] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ==================== 17:17:41 (1476490661) [ 9766.111123] Lustre: DEBUG MARKER: SKIP: sanity test_27n too few OSTs [ 9767.282520] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ====== 17:17:42 (1476490662) [ 9767.680506] Lustre: DEBUG MARKER: SKIP: sanity test_27o too few OSTs [ 9768.822170] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ===== 17:17:44 (1476490664) [ 9769.210744] Lustre: DEBUG MARKER: SKIP: sanity test_27p too few OSTs [ 9770.350202] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ===== 17:17:45 (1476490665) [ 9770.731822] Lustre: DEBUG MARKER: SKIP: sanity test_27q too few OSTs [ 9771.877991] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) === 17:17:47 (1476490667) [ 9772.265388] Lustre: DEBUG MARKER: SKIP: sanity test_27r too few OSTs [ 9773.494591] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) == 17:17:49 (1476490669) [ 9774.648700] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly == 17:17:50 (1476490670) [ 9775.801332] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ============ 17:17:51 (1476490671) [ 9776.172160] Lustre: DEBUG MARKER: SKIP: sanity test_27u too few OSTs [ 9777.329338] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST =================== 17:17:52 (1476490672) [ 9777.723414] Lustre: DEBUG MARKER: SKIP: sanity test_27v too few OSTs [ 9778.882217] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option == 17:17:54 (1476490674) [ 9780.046595] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options == 17:17:55 (1476490675) [ 9780.422842] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [ 9781.586548] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 17:17:57 (1476490677) [ 9781.981520] Lustre: DEBUG MARKER: SKIP: sanity test_27x 1 < 2 OSTs [ 9783.135624] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive == 17:17:58 (1476490678) [ 9783.533349] Lustre: DEBUG MARKER: SKIP: sanity test_27y 1 < 2 OSTs -- skipping [ 9784.702205] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems == 17:18:00 (1476490680) [ 9790.258225] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 9790.656978] Lustre: DEBUG MARKER: FID seq 0x200007161, oid 0x1a0fc ver 0x0 [ 9791.012357] Lustre: DEBUG MARKER: LOV seq 0x200007161, oid 0x1a0fc, count: 1 [ 9791.417437] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:103909/0x195e5 seq:0 [ 9792.081857] Lustre: Failing over lustre-OST0000 [ 9792.596621] Lustre: server umount lustre-OST0000 complete [ 9793.654095] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 9793.701784] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [ 9795.642407] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 9795.669350] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 9795.887865] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.503749] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 9801.868695] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 9801.882266] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [ 9801.896738] Lustre: Skipped 5 previous similar messages [ 9806.049208] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 9806.083024] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103911 to 0x0:103937 [ 9806.562973] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 9806.959030] Lustre: DEBUG MARKER: FID seq 0x200007161, oid 0x1a0fe ver 0x0 [ 9807.334140] Lustre: DEBUG MARKER: LOV seq 0x200007161, oid 0x1a0fe, count: 1 [ 9807.706697] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:103910/0x195e6 seq:0 [ 9808.381746] Lustre: Failing over lustre-OST0000 [ 9809.199161] Lustre: server umount lustre-OST0000 complete [ 9810.191585] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 9810.239196] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [ 9812.055742] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 9812.083565] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 9812.885932] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 9815.514946] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 9819.990942] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 9820.024785] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103911 to 0x0:103969 [ 9821.258540] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values == 17:18:36 (1476490716) [ 9822.508993] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 17:18:38 (1476490718) [ 9823.708027] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs == 17:18:39 (1476490719) [ 9824.072997] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [ 9825.253861] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API == 17:18:40 (1476490720) [ 9825.628361] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [ 9826.761089] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases == 17:18:42 (1476490722) [ 9827.150404] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [ 9828.297138] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size == 17:18:43 (1476490723) [ 9829.330006] Lustre: Failing over lustre-OST0000 [ 9830.056966] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.266125] Lustre: server umount lustre-OST0000 complete [ 9831.617405] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [ 9831.648236] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [ 9832.448868] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 [ 9839.920436] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 3 clients reconnect [ 9842.280373] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 9842.314240] Lustre: lustre-OST0000: deleting orphan objects from 0x0:103976 to 0x0:104001 [ 9845.994723] Lustre: DEBUG MARKER: sanity test_27F: @@@@@@ FAIL: wrong stripecount [ 9848.155562] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============== 17:19:03 (1476490743) [ 9849.435439] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================== 17:19:04 (1476490744) [ 9849.836192] Lustre: DEBUG MARKER: first d29 [ 9850.242735] Lustre: DEBUG MARKER: second d29 [ 9850.632973] Lustre: DEBUG MARKER: done [ 9851.799580] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ================ 17:19:07 (1476490747) [ 9852.950355] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ============= 17:19:08 (1476490748) [ 9854.118248] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ====== 17:19:09 (1476490749) [ 9855.381780] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ==================================== 17:19:10 (1476490750) [ 9856.559631] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ========= 17:19:12 (1476490752) [ 9857.713702] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links =============== 17:19:13 (1476490753) [ 9858.887982] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =========================== 17:19:14 (1476490754) [ 9860.025516] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ================= 17:19:15 (1476490755) [ 9861.186673] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ===== 17:19:16 (1476490756) [ 9868.436648] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================= 17:19:23 (1476490763) [ 9869.601442] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================= 17:19:25 (1476490765) [ 9870.788074] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================= 17:19:26 (1476490766) [ 9871.962277] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================= 17:19:27 (1476490767) [ 9873.109337] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir================= 17:19:28 (1476490768) [ 9874.284536] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir================= 17:19:29 (1476490769) [ 9875.428087] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file == 17:19:30 (1476490770) [ 9876.608536] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename == 17:19:32 (1476490772) [ 9884.966230] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory == 17:19:40 (1476490780) [ 9885.337177] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 9886.475669] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ======================= 17:19:42 (1476490782) [ 9887.673737] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ======================= 17:19:43 (1476490783) [ 9888.857247] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir =========== 17:19:44 (1476490784) [ 9890.022742] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir =========== 17:19:45 (1476490785) [ 9891.213029] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ====== 17:19:46 (1476490786) [ 9892.412315] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ====== 17:19:47 (1476490787) [ 9893.591282] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 == 17:19:49 (1476490789) [ 9894.785550] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 == 17:19:50 (1476490790) [ 9895.964082] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file ============= 17:19:51 (1476490791) [ 9897.140374] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file ============= 17:19:52 (1476490792) [ 9898.327049] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ========== 17:19:53 (1476490793) [ 9899.522415] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ========== 17:19:55 (1476490795) [ 9900.704079] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ======== 17:19:56 (1476490796) [ 9901.891111] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ======== 17:19:57 (1476490797) [ 9903.104413] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ == 17:19:58 (1476490798) [ 9904.288729] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ == 17:19:59 (1476490799) [ 9904.678773] Lustre: DEBUG MARKER: 32p_1 [ 9905.068397] Lustre: DEBUG MARKER: 32p_2 [ 9905.433715] Lustre: DEBUG MARKER: 32p_3 [ 9905.812516] Lustre: DEBUG MARKER: 32p_4 [ 9906.220857] Lustre: DEBUG MARKER: 32p_5 [ 9906.600978] Lustre: DEBUG MARKER: 32p_6 [ 9906.956745] Lustre: DEBUG MARKER: 32p_7 [ 9907.350892] Lustre: DEBUG MARKER: 32p_8 [ 9907.732787] Lustre: DEBUG MARKER: 32p_9 [ 9908.126156] Lustre: DEBUG MARKER: 32p_10 [ 9909.269343] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) == 17:20:04 (1476490804) [ 9910.424230] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) == 17:20:05 (1476490805) [ 9911.636296] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ====== 17:20:07 (1476490807) [ 9912.379925] Lustre: DEBUG MARKER: 33_1 [ 9912.743785] Lustre: DEBUG MARKER: 33_2 [ 9913.894963] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) == 17:20:09 (1476490809) [ 9915.212314] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) == 17:20:10 (1476490810) [ 9916.564744] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes == 17:20:12 (1476490812) [ 9918.116456] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir == 17:20:13 (1476490813) [ 9918.509861] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 9919.642051] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode == 17:20:15 (1476490815) [ 9920.027747] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 9921.165728] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory == 17:20:16 (1476490816) [ 9921.535872] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 9922.681246] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ============= 17:20:18 (1476490818) [ 9923.850321] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ======= 17:20:19 (1476490819) [ 9925.013923] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ================ 17:20:20 (1476490820) [ 9926.164482] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file =============================== 17:20:21 (1476490821) [ 9927.312430] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ==== 17:20:22 (1476490822) [ 9928.499683] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ========= 17:20:24 (1476490824) [ 9929.700345] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ================================= 17:20:25 (1476490825) [ 9930.898191] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 17:20:26 (1476490826) [ 9934.793703] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ======= 17:20:30 (1476490830) [ 9936.088122] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ===================== 17:20:31 (1476490831) [ 9937.246568] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ====================== 17:20:32 (1476490832) [ 9938.363871] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ============ 17:20:33 (1476490833) [ 9939.715174] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ============= 17:20:35 (1476490835) [ 9940.899405] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ====== 17:20:36 (1476490836) [ 9942.090515] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ============ 17:20:37 (1476490837) [ 9944.297507] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ======================= 17:20:39 (1476490839) [10068.536817] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ============ 17:22:43 (1476490963) [10070.735346] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory == 17:22:46 (1476490966) [10071.121658] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [10072.310744] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ===== 17:22:47 (1476490967) [10073.491540] Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create ============================= 17:22:48 (1476490968) [10076.679177] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ======== 17:22:52 (1476490972) [10078.999380] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ============================= 17:22:54 (1476490974) [10082.234710] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ================================ 17:22:57 (1476490977) [10083.431088] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ========================== 17:22:58 (1476490978) [10084.631901] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =================== 17:23:00 (1476490980) [10087.863723] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ================================= 17:23:03 (1476490983) [10091.051002] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat =============== 17:23:06 (1476490986) [10093.602966] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ================================ 17:23:09 (1476490989) [10095.773358] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ========================= 17:23:11 (1476490991) [10099.050245] Lustre: 2 MB is too small for debug buffer size, setting it to 128 MB. [10100.570601] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ========================== 17:23:16 (1476490996) [10102.927899] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ============================= 17:23:18 (1476490998) [10103.748295] Lustre: DEBUG MARKER: sanity test_39l: @@@@@@ FAIL: atime is not updated from future: 1508026998, 1476490999= 2 MDTs [10124.201451] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate =============== 17:23:39 (1476491019) [10125.458688] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ======================= 17:23:40 (1476491020) [10127.636272] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [10128.036064] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [10128.447334] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [10128.830967] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [10129.234224] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously == 17:23:44 (1476491024) [10148.158447] Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY == 17:24:03 (1476491043) [10150.364555] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY == 17:24:05 (1476491045) [10151.597246] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY == 17:24:07 (1476491047) [10153.250561] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre========================== 17:24:08 (1476491048) [10154.439825] Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe =============== 17:24:09 (1476491049) [10154.846365] Lustre: DEBUG MARKER: SKIP: sanity test_44 skipping 2-stripe test [10156.014103] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ================================= 17:24:11 (1476491051) [10157.502312] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [10157.898706] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================== 17:24:13 (1476491053) [10159.287311] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=== 17:24:14 (1476491054) [10160.557362] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=== 17:24:15 (1476491055) [10161.751771] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) == 17:24:17 (1476491057) [10162.959608] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) == 17:24:18 (1476491058) [10164.171362] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) == 17:24:19 (1476491059) [10165.390382] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent == 17:24:20 (1476491060) [10166.870529] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ================= 17:24:22 (1476491062) [10168.087662] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ==== 17:24:23 (1476491063) [10169.686586] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit == 17:24:25 (1476491065) [10311.516813] Lustre: DEBUG MARKER: == sanity test 51ba: verify nlink for many subdirectory cleanup == 17:26:46 (1476491206) [10621.450777] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d [10621.826883] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit == 17:31:57 (1476491517) [11165.793221] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ======= 17:41:00 (1476492060) [11167.144797] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ========= 17:41:02 (1476492062) [11168.444151] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ====== 17:41:03 (1476492063) [11172.281718] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ============================ 17:41:07 (1476492067) [11173.524681] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ======================== 17:41:08 (1476492068) [11174.696022] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ======================= 17:41:09 (1476492069) [11177.265873] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ======================== 17:41:12 (1476492072) [11178.403536] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ======================== 17:41:13 (1476492073) [11179.615817] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe == 17:41:14 (1476492074) [11180.262888] Lustre: DEBUG MARKER: SKIP: sanity test_56a skipping other /usr/bin/lfs getstripe --obd test [11181.465471] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name =============================== 17:41:16 (1476492076) [11182.946903] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============================== 17:41:17 (1476492077) [11184.380502] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========= 17:41:19 (1476492079) [11185.573902] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============================== 17:41:20 (1476492080) [11187.003100] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============================== 17:41:21 (1476492081) [11188.458910] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============================== 17:41:23 (1476492083) [11189.900155] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============================== 17:41:24 (1476492084) [11191.337668] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============================== 17:41:26 (1476492086) [11192.770493] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================ 17:41:27 (1476492087) [11194.351710] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ================================= 17:41:29 (1476492089) [11195.975101] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ================================= 17:41:30 (1476492090) [11197.454382] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ============================ 17:41:32 (1476492092) [11199.307699] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works == 17:41:34 (1476492094) [11200.894485] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works == 17:41:35 (1476492095) [11202.616216] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works == 17:41:37 (1476492097) [11204.273895] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ========= 17:41:39 (1476492099) [11205.896985] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works == 17:41:40 (1476492100) [11206.277890] Lustre: DEBUG MARKER: SKIP: sanity test_56w 1 < 2 OSTs -- skipping [11207.437613] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support == 17:41:42 (1476492102) [11207.835402] Lustre: DEBUG MARKER: SKIP: sanity test_56x need 2 OST, skipping test [11209.000321] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released == 17:41:43 (1476492103) [11210.590261] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error == 17:41:45 (1476492105) [11213.034948] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir == 17:41:47 (1476492107) [11213.396123] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [11214.558408] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ==== 17:41:49 (1476492109) [11216.688287] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ===== 17:41:51 (1476492111) [11221.798520] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ================ 17:41:56 (1476492116) [11222.968786] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =========== 17:41:57 (1476492117) [11238.523948] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============ 17:42:13 (1476492133) [11239.233450] Lustre: DEBUG MARKER: test_60 run 2099 - from kernel mode [11275.375485] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [11275.388406] Lustre: Skipped 8 previous similar messages [11278.396648] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========== 17:42:53 (1476492173) [11282.961949] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [11282.961986] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [11282.963137] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361fe4ec65f to 0x20295361fef7ba39 [11283.018777] LustreError: Skipped 1 previous similar message [11283.028396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [11283.053987] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11297.538055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:105846 to 0x0:105889 [11298.782385] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full == 17:43:13 (1476492193) [11338.960646] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 17:43:53 (1476492233) [11340.148989] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created == 17:43:54 (1476492234) [11341.684930] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================== 17:43:56 (1476492236) [11342.945808] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 17:43:57 (1476492237) [11404.214244] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ===== 17:44:59 (1476492299) [11408.082435] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ======= 17:45:02 (1476492302) [11409.259668] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [11409.654556] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ == 17:45:04 (1476492304) [11410.829281] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ====================== 17:45:05 (1476492305) [11412.207658] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 == 17:45:07 (1476492307) [11413.435624] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 == 17:45:08 (1476492308) [11414.579633] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count == 17:45:09 (1476492309) [11415.831632] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========================= 17:45:10 (1476492310) [11417.068491] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ===== 17:45:11 (1476492311) [11418.272887] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d ============================= 17:45:13 (1476492313) [11419.505683] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ====================== 17:45:14 (1476492314) [11420.736161] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)=== 17:45:15 (1476492315) [11421.935316] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping == 17:45:16 (1476492316) [11424.165442] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping == 17:45:18 (1476492318) [11425.833541] Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping == 17:45:20 (1476492320) [11438.481567] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=== 17:45:33 (1476492333) [11440.734775] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs == 17:45:35 (1476492335) [11441.128778] Lustre: DEBUG MARKER: SKIP: sanity test_65k too few OSTs [11442.287723] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ========================== 17:45:37 (1476492337) [11443.537727] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ================= 17:45:38 (1476492338) [11446.605574] Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) [11447.047436] Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b [11447.466014] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ======== 17:45:42 (1476492342) [11448.432462] Lustre: *** cfs_fail_loc=217, val=0*** [11448.441029] Lustre: Skipped 3 previous similar messages [11450.065236] Lustre: *** cfs_fail_loc=217, val=0*** [11450.073712] Lustre: Skipped 3 previous similar messages [11451.616936] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [11452.009324] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ====== 17:45:46 (1476492346) [11453.466499] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) == 17:45:48 (1476492348) [11455.304020] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) == 17:45:50 (1476492350) [11482.583620] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) == 17:46:17 (1476492377) [11483.770622] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) == 17:46:18 (1476492378) [11484.965969] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) == 17:46:19 (1476492379) [11486.145733] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [11486.530177] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation == 17:46:21 (1476492381) [11488.965743] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read == 17:46:23 (1476492383) [11489.070107] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200008101:0xd1b2:0x0] object 0x0:110915 extent [0-1048575]: client csum 178f51eb, server csum 178f51ea [11490.803572] Lustre: DEBUG MARKER: set checksum type to crc32 [11492.847348] Lustre: DEBUG MARKER: set checksum type to adler [11494.577133] Lustre: DEBUG MARKER: set checksum type to adler [11495.746481] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read == 17:46:30 (1476492390) [11495.871995] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200008101:0xd1b3:0x0] object 0x0:110916 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [11499.217512] Lustre: DEBUG MARKER: SKIP: sanity test_77f skipping SLOW test 77f [11499.624791] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read == 17:46:34 (1476492394) [11500.079258] Lustre: *** cfs_fail_loc=21a, val=0*** [11500.088934] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200008101:0xd1b5:0x0] object 0x0:110917 extent [0-1048575]: client csum 178f51ea, server csum d9be5095 [11501.778664] Lustre: *** cfs_fail_loc=21b, val=0*** [11504.540763] Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM == 17:46:39 (1476492399) [11505.043189] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [11505.057839] Lustre: Skipped 2 previous similar messages [11506.378227] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 == 17:46:41 (1476492401) [11509.898619] Lustre: DEBUG MARKER: SKIP: sanity test_78 skipping SLOW test 78 [11510.300063] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ========================= 17:46:45 (1476492405) [11512.849195] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ====== 17:46:47 (1476492407) [11515.541897] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ================= 17:46:50 (1476492410) [11515.956861] Lustre: *** cfs_fail_loc=228, val=0*** [11515.965272] Lustre: Skipped 3 previous similar messages [11517.724246] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ========= 17:46:52 (1476492412) [11518.113327] Lustre: *** cfs_fail_loc=228, val=0*** [11520.026591] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ================================= 17:46:54 (1476492414) [11523.726177] Lustre: DEBUG MARKER: == sanity test 99a: cvs init =========================================== 17:46:58 (1476492418) [11526.021402] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ========================================= 17:47:00 (1476492420) [11528.281858] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ======================================= 17:47:03 (1476492423) [11532.043254] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ============================================ 17:47:06 (1476492426) [11533.271639] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ========================================= 17:47:07 (1476492427) [11534.703371] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ========================================= 17:47:09 (1476492429) [11537.229516] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ============= 17:47:11 (1476492431) [11538.694180] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads ================== 17:47:13 (1476492433) [11558.612025] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =================== 17:47:33 (1476492453) [11559.008483] Lustre: DEBUG MARKER: SKIP: sanity test_101b skipping stride IO stride-ahead test [11560.184670] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead =================== 17:47:34 (1476492454) [11584.158325] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled == 17:47:58 (1476492478) [11792.016243] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) == 17:51:26 (1476492686) [11803.432809] Lustre: DEBUG MARKER: == sanity test 101f: check read-ahead for max_read_ahead_whole_mb == 17:51:38 (1476492698) [11804.764923] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ==================================== 17:51:39 (1476492699) [11806.060928] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ============== 17:51:40 (1476492700) [11806.442574] Lustre: DEBUG MARKER: SKIP: sanity test_102b skipping 2-stripe test [11807.610010] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ============= 17:51:42 (1476492702) [11807.996559] Lustre: DEBUG MARKER: SKIP: sanity test_102c skipping 2-stripe test [11809.170165] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============= 17:51:43 (1476492703) [11809.570475] Lustre: DEBUG MARKER: SKIP: sanity test_102d skipping N-stripe test [11810.726194] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ============= 17:51:45 (1476492705) [11811.103385] Lustre: DEBUG MARKER: SKIP: sanity test_102f skipping N-stripe test [11812.295059] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block == 17:51:46 (1476492706) [11812.690902] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [11813.104494] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [11813.509538] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [11813.907326] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [11815.068620] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode == 17:51:49 (1476492709) [11815.780712] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [11816.960008] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ============== 17:51:51 (1476492711) [11818.150448] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ===== 17:51:52 (1476492712) [11818.554691] Lustre: DEBUG MARKER: SKIP: sanity test_102j skipping N-stripe test [11819.739339] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash == 17:51:54 (1476492714) [11821.059051] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ==================================== 17:51:55 (1476492715) [11822.274102] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========== 17:51:56 (1476492716) [11823.464128] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs == 17:51:58 (1476492718) [11825.421927] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission == 17:52:00 (1476492720) [11826.967160] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans == 17:52:01 (1476492721) [11828.448680] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values == 17:52:03 (1476492723) [11830.040909] Lustre: DEBUG MARKER: == sanity test 103a: acl test =========================================== 17:52:04 (1476492724) [11846.121401] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' == 17:52:20 (1476492740) [11855.822755] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [11855.835494] Lustre: Skipped 3 previous similar messages [11858.241068] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [11858.241189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [11858.242203] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11858.301888] LustreError: Skipped 1 previous similar message [11858.311335] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [11858.333248] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361fef7ba39 to 0x20295361ff01bcbe [11880.112324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:112241 to 0x0:112321 [11883.253189] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [11883.268372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [11893.493284] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl == 17:53:08 (1476492788) [11908.265376] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [11908.279616] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [11908.301811] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff01bcbe to 0x20295361ff01bf3b [11918.734795] Lustre: lustre-OST0000: deleting orphan objects from 0x0:112323 to 0x0:112353 [11920.125301] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =========================== 17:53:34 (1476492814) [11920.256549] Lustre: lustre-OST0000: Client 4050fac7-8562-b560-a503-e4332c6e3720 (at 10.2.4.36@tcp) reconnecting [11920.696201] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88004a2d0400.ost_server_uuid in FULL state after 0 sec [11921.915336] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================== 17:53:36 (1476492816) [11923.123528] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ========== 17:53:37 (1476492817) [11924.353930] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ========== 17:53:38 (1476492818) [11925.544557] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========== 17:53:40 (1476492820) [11926.795545] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ========== 17:53:41 (1476492821) [11933.299785] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11938.078073] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========= 17:53:52 (1476492832) [11939.302195] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir == 17:53:53 (1476492833) [11940.511249] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG == 17:53:55 (1476492835) [11942.745411] Lustre: DEBUG MARKER: == sanity test 110: filename length checking == 17:53:57 (1476492837) [11944.058056] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [11944.477542] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ===================== 17:53:58 (1476492838) [11944.845342] Lustre: DEBUG MARKER: SKIP: sanity test_116a 1 < 2 OSTs [11946.038803] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass == 17:54:00 (1476492840) [11948.957157] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ============ 17:54:03 (1476492843) [11950.194065] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============ 17:54:04 (1476492844) [11951.794264] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============ 17:54:06 (1476492846) [11952.298001] Lustre: *** cfs_fail_loc=217, val=0*** [11954.257321] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============ 17:54:08 (1476492848) [11954.780196] LustreError: 17021:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 sleeping for 1000ms [11955.795373] LustreError: 17021:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 awake [11958.437487] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ============ 17:54:12 (1476492852) [11958.907763] LustreError: 17021:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 25000ms [11983.935056] LustreError: 17021:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [11985.966794] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ============ 17:54:40 (1476492880) [11987.602919] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ============ 17:54:42 (1476492882) [11989.201782] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ============ 17:54:43 (1476492883) [11989.671851] Lustre: *** cfs_fail_loc=20e, val=0*** [11992.690785] Lustre: *** cfs_fail_loc=20e, val=0*** [11992.699474] Lustre: Skipped 1 previous similar message [11999.713416] Lustre: *** cfs_fail_loc=20e, val=0*** [11999.722009] Lustre: Skipped 1 previous similar message [12001.435374] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ============ 17:54:55 (1476492895) [12009.739344] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ============ 17:55:04 (1476492904) [12010.304129] Lustre: *** cfs_fail_loc=220, val=0*** [12010.312698] Lustre: Skipped 3 previous similar messages [12012.043507] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =========== 17:55:06 (1476492906) [12022.333741] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========== 17:55:16 (1476492916) [12023.941635] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =========== 17:55:18 (1476492918) [12025.350699] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount == 17:55:19 (1476492919) [12027.030141] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount == 17:55:21 (1476492921) [12027.434404] Lustre: DEBUG MARKER: SKIP: sanity test_119b skipping 2-stripe test [12028.601583] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole == 17:55:23 (1476492923) [12030.088307] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed == 17:55:24 (1476492924) [12031.871687] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [12035.025402] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test == 17:55:29 (1476492929) [12036.853241] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test == 17:55:31 (1476492931) [12038.750519] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test == 17:55:33 (1476492933) [12040.573401] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test == 17:55:35 (1476492935) [12042.385827] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test == 17:55:36 (1476492936) [12049.293414] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test == 17:55:43 (1476492943) [12056.237923] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test == 17:55:50 (1476492950) [12080.627924] Lustre: DEBUG MARKER: == sanity test 121: read cancel race =========== 17:56:15 (1476492975) [12082.075983] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work == 17:56:16 (1476492976) [12082.472268] Lustre: DEBUG MARKER: testing on UP system. Performance may be not as good as expected. [12083.076689] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [12083.599283] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [12086.682091] Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec [12087.761578] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [12114.873164] Lustre: DEBUG MARKER: ls 10000 files without statahead: 16 sec [12122.041163] Lustre: DEBUG MARKER: ls 10000 files with statahead: 6 sec [12122.442786] Lustre: DEBUG MARKER: ls done [12130.405052] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 7 seconds [12130.805637] Lustre: DEBUG MARKER: rm done [12131.942580] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) == 17:57:06 (1476493026) [12136.789536] Lustre: DEBUG MARKER: ls done [12139.073245] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ========================================= 17:57:13 (1476493033) [12139.409942] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [12144.534263] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88004a2d0400 [12144.932558] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88004a2d0400 [12145.316834] Lustre: DEBUG MARKER: LRU=2003 [12145.716647] Lustre: DEBUG MARKER: LIMIT=1596099 [12146.101544] Lustre: DEBUG MARKER: LVF=956225 [12146.496087] Lustre: DEBUG MARKER: OLD_LVF=1 [12146.869438] Lustre: DEBUG MARKER: Sleep 50 sec [12197.328198] Lustre: DEBUG MARKER: Dropped 668 locks in 50s [12197.708614] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [12200.320623] Lustre: DEBUG MARKER: SKIP: sanity test_124b skipping SLOW test 124b [12200.716743] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks == 17:58:15 (1476493095) [12222.653238] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs == 17:58:37 (1476493117) [12224.025021] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account == 17:58:38 (1476493118) [12225.224662] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane == 17:58:39 (1476493119) [12227.195850] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane == 17:58:41 (1476493121) [12228.420923] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's == 17:58:42 (1476493122) [12229.716457] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ========================== 17:58:44 (1476493124) [12330.262558] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) == 18:00:24 (1476493224) [12331.546203] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) == 18:00:25 (1476493225) [12331.963423] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping FIEMAP on 2-stripe file test [12333.187774] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) == 18:00:27 (1476493227) [12333.593804] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping FIEMAP on 2-stripe file [12334.818166] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) == 18:00:29 (1476493229) [12335.228283] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [12336.441818] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) == 18:00:30 (1476493230) [12336.847279] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping continuation FIEMAP test [12338.065754] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv == 18:00:32 (1476493232) [12339.485531] Lustre: DEBUG MARKER: == sanity test 131b: test append writev == 18:00:33 (1476493233) [12340.848658] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects == 18:00:35 (1476493235) [12342.086205] Lustre: DEBUG MARKER: == sanity test 131d: test short read == 18:00:36 (1476493236) [12343.327391] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole == 18:00:37 (1476493237) [12344.522656] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ========================================== 18:00:38 (1476493238) [12350.135146] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ==================================== 18:00:44 (1476493244) [12353.735323] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ========================================== 18:00:48 (1476493248) [12372.304364] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================== 18:01:06 (1476493266) [12378.338254] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =================== 18:01:12 (1476493272) [12381.438822] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 18:01:15 (1476493275) [12386.718240] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc == 18:01:21 (1476493281) [12389.095590] LustreError: 48691:0:(ldlm_resource.c:109:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [12389.110095] LustreError: 48691:0:(ldlm_resource.c:109:seq_watermark_write()) Skipped 9 previous similar messages [12389.167611] LustreError: 48723:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Can't parse user input, rc = -14 [12389.181401] LustreError: 48723:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Skipped 14 previous similar messages [12389.209159] LustreError: 48733:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) invalid value written [12389.222990] LustreError: 48733:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) Skipped 7 previous similar messages [12389.384746] LustreError: 48813:0:(genops.c:1519:obd_export_evict_by_uuid()) lustre-OST0000: can't disconnect : no exports found [12389.401204] LustreError: 48813:0:(genops.c:1519:obd_export_evict_by_uuid()) Skipped 3 previous similar messages [12389.559266] LustreError: 48942:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) srv-lustre-OST0000: invalid FID sequence width: rc = -14 [12389.578137] LustreError: 48942:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) Skipped 4 previous similar messages [12389.599115] LustreError: 48948:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) cannot add nodemap config to non-existing MGS. [12389.616342] LustreError: 48948:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) Skipped 1 previous similar message [12389.638568] LustreError: 48956:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [12389.674517] Lustre: 48973:0:(libcfs_string.c:131:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [12389.693529] Lustre: 48973:0:(libcfs_string.c:131:cfs_str2mask()) Skipped 2 previous similar messages [12389.708847] Lustre: 0 MB is too small for debug buffer size, setting it to 128 MB. [12409.693617] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [12409.706227] Lustre: Skipped 6 previous similar messages [12412.526614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12415.510332] Lustre: 23244:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476493303/real 1476493303] req@ffff88041a8da700 x1548202901273264/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476493310 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12415.551751] Lustre: 23244:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12415.565869] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [12415.591215] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [12415.591356] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [12415.627509] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [12415.650326] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff01bf3b to 0x20295361ff21dfe0 [12434.658943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:137050 to 0x0:137217 [12440.602424] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [12440.617701] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [12458.088194] Lustre: server umount lustre-OST0000 complete [12481.338274] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [12481.366728] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [12483.718958] Lustre: lustre-OST0000: deleting orphan objects from 0x0:137050 to 0x0:137249 [12486.506084] Lustre: DEBUG MARKER: Using TIMEOUT=100 [12493.057709] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold == 18:03:07 (1476493387) [12509.677385] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb == 18:03:23 (1476493403) [12618.475858] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ====== 18:05:12 (1476493512) [12620.052991] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests == 18:05:14 (1476493514) [12622.147392] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ================================= 18:05:16 (1476493516) [12628.488885] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ============================== 18:05:22 (1476493522) [12629.738098] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ========================= 18:05:23 (1476493523) [12631.184705] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks == 18:05:25 (1476493525) [12632.766224] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID == 18:05:26 (1476493526) [12635.648271] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory == 18:05:29 (1476493529) [12636.365976] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [12637.513917] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments == 18:05:31 (1476493531) [12639.194604] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid == 18:05:33 (1476493533) [12641.146930] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 18:05:35 (1476493535) [12642.632304] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 18:05:36 (1476493536) [12643.973943] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests == 18:05:38 (1476493538) [12704.704783] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on == 18:06:38 (1476493598) [12707.001523] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off == 18:06:41 (1476493601) [12709.075626] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on == 18:06:43 (1476493603) [12711.181956] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off == 18:06:45 (1476493605) [12713.247439] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on == 18:06:47 (1476493607) [12936.603437] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off == 18:10:30 (1476493830) [13165.856168] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on == 18:14:19 (1476494059) [13406.397043] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off == 18:18:20 (1476494300) [13640.108757] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables == 18:22:13 (1476494533) [13641.848724] Lustre: DEBUG MARKER: Turn on read and write cache [13642.929351] Lustre: DEBUG MARKER: Write data and read it back. [13643.315254] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [13644.463138] Lustre: DEBUG MARKER: cache hits:: before: 655392, after: 655395 [13644.850756] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [13645.635436] Lustre: DEBUG MARKER: cache hits:: before: 655395, after: 655398 [13646.007145] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [13647.107364] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [13648.239443] Lustre: DEBUG MARKER: cache hits:: before: 655398, after: 655401 [13648.628865] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [13649.458990] Lustre: DEBUG MARKER: cache hits:: before: 655401, after: 655401 [13649.841432] Lustre: DEBUG MARKER: Write data and read it back. [13650.227907] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [13651.411857] Lustre: DEBUG MARKER: cache hits:: before: 655401, after: 655404 [13651.793231] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [13652.627582] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [13653.021234] Lustre: DEBUG MARKER: Turn off read and write cache [13654.141816] Lustre: DEBUG MARKER: Write data and read it back [13654.525828] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [13655.771543] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [13656.160077] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [13657.272930] Lustre: DEBUG MARKER: Write data and read it back [13657.672895] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [13658.866180] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655404 [13659.241728] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [13660.389860] Lustre: DEBUG MARKER: cache hits:: before: 655404, after: 655407 [13661.553115] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity == 18:22:35 (1476494555) [13676.566241] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [13676.578280] Lustre: Skipped 6 previous similar messages [13682.399509] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [13682.399535] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [13682.400478] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff21e1a0 to 0x20295361ff2aa1c3 [13682.452281] LustreError: Skipped 1 previous similar message [13682.461083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [13682.485278] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13684.973937] Lustre: lustre-OST0000: deleting orphan objects from 0x0:138891 to 0x0:139041 [13686.531789] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog == 18:23:00 (1476494580) [13689.268837] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event == 18:23:02 (1476494582) [13692.884265] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event == 18:23:06 (1476494586) [13693.256586] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [13694.411573] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity == 18:23:08 (1476494588) [13699.935485] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory == 18:23:13 (1476494593) [13700.329004] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [13701.502673] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags == 18:23:15 (1476494595) [13703.950000] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity == 18:23:17 (1476494597) [13705.297535] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity == 18:23:18 (1476494598) [13705.696748] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [13706.848988] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep == 18:23:20 (1476494600) [13712.656288] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock == 18:23:26 (1476494606) [13713.029583] Lustre: DEBUG MARKER: creating a 10 Mb file [13714.255916] Lustre: DEBUG MARKER: starting reads [13714.879851] Lustre: DEBUG MARKER: truncating the file [13715.273100] Lustre: DEBUG MARKER: killing dd [13715.650345] Lustre: DEBUG MARKER: removing the temporary file [13716.809334] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ======================= 18:23:30 (1476494610) [13720.012423] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ======== 18:23:33 (1476494613) [13724.289456] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc == 18:23:37 (1476494617) [13724.358882] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [13724.373213] Lustre: Skipped 2 previous similar messages [13727.252520] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 18:23:40 (1476494620) [13727.626598] Lustre: Echo OBD driver; http://www.lustre.org/ [13733.001450] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT == 18:23:46 (1476494626) [13733.865306] Lustre: Echo OBD driver; http://www.lustre.org/ [13746.844362] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ========================== 18:24:00 (1476494640) [13763.119350] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ================== 18:24:16 (1476494656) [13772.191722] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========== 18:24:25 (1476494665) [13774.509793] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap == 18:24:28 (1476494668) [13775.949650] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) == 18:24:29 (1476494669) [13777.246220] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap == 18:24:30 (1476494670) [13802.105235] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap == 18:24:55 (1476494695) [13803.389077] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps == 18:24:57 (1476494697) [13805.000786] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support == 18:24:58 (1476494698) [13807.699986] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change == 18:25:01 (1476494701) [13811.433122] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file == 18:25:05 (1476494705) [13813.487331] Lustre: DEBUG MARKER: == sanity test 200: OST pools == 18:25:07 (1476494707) [13852.915049] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes =================== 18:25:46 (1476494746) [13854.218624] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ============= 18:25:47 (1476494747) [13855.425857] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ============= 18:25:49 (1476494749) [13856.626827] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size =============== 18:25:50 (1476494750) [13857.865813] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes =================== 18:25:51 (1476494751) [13859.119979] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ============= 18:25:52 (1476494752) [13860.390244] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset ============= 18:25:53 (1476494753) [13861.614653] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size =============== 18:25:55 (1476494755) [13862.806549] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats == 18:25:56 (1476494756) [13873.733530] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [13874.114197] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.24969 [13874.892336] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [13875.272712] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.3136 [13875.994558] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [13876.378003] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.22985 [13877.168901] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [13877.565777] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.22550 [13878.292477] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [13878.670171] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.15638 [13879.413250] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [13879.818392] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.7273 [13880.862711] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [13881.254150] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.20139 [13882.474107] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [13882.868326] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.19996 [13883.963189] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [13884.350604] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.13373 [13885.402464] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [13885.795503] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.20254 [13886.588891] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [13886.977354] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.26352 [13907.712343] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug == 18:26:41 (1476494801) [13908.914994] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse == 18:26:42 (1476494802) [13910.169585] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open == 18:26:43 (1476494803) [13911.520273] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open == 18:26:45 (1476494805) [13935.112557] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [13939.523260] Lustre: 23238:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476494826/real 1476494826] req@ffff88041a8df800 x1548202901279136/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476494833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13939.523293] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [13939.524231] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff2aa1c3 to 0x20295361ff3e8bde [13939.601671] Lustre: 23238:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [13939.615430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [13939.639781] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13947.543324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153089 to 0x0:153729 [13948.080062] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [13971.653810] Lustre: 23239:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476494858/real 1476494858] req@ffff88041a8de900 x1548202901279440/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476494865 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13971.693908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [13971.718886] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13989.116414] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153089 to 0x0:153761 [13990.307709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 17 sec [13992.746782] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly == 18:28:06 (1476494886) [13996.730756] LustreError: 11-0: MGC10.2.4.47@tcp: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [13996.744786] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [13996.766861] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff3e8bde to 0x20295361ff3e8f3b [14001.497085] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ============================================== 18:28:15 (1476494895) [14003.220321] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 == 18:28:16 (1476494896) [14004.532655] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 == 18:28:18 (1476494898) [14008.408907] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 == 18:28:21 (1476494901) [14009.658469] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly == 18:28:23 (1476494903) [14013.735704] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') == 18:28:27 (1476494907) [14015.926212] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================= 18:28:29 (1476494909) [14016.336734] Lustre: DEBUG MARKER: creating a 10 Mb file [14017.599874] Lustre: DEBUG MARKER: starting reads [14018.383577] Lustre: DEBUG MARKER: truncating the file [14018.785623] Lustre: DEBUG MARKER: killing dd [14019.176071] Lustre: DEBUG MARKER: removing the temporary file [14020.373913] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND == 18:28:33 (1476494913) [14022.008070] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST == 18:28:35 (1476494915) [14029.226436] Lustre: *** cfs_fail_loc=229, val=0*** [14029.235256] Lustre: Skipped 1 previous similar message [14034.252926] Lustre: *** cfs_fail_loc=229, val=0*** [14039.279349] Lustre: *** cfs_fail_loc=229, val=0*** [14044.305784] Lustre: *** cfs_fail_loc=229, val=0*** [14056.786595] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM == 18:29:10 (1476494950) [14084.395656] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================== 18:29:37 (1476494977) [14085.870842] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure =============== 18:29:39 (1476494979) [14087.136575] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ========================= 18:29:40 (1476494980) [14088.416884] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure == 18:29:41 (1476494981) [14088.473827] Lustre: lustre-OST0000: Client d158399d-7705-3049-2b52-0cf9ec22ab30 (at 10.2.4.36@tcp) reconnecting [14088.487938] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [14088.502656] Lustre: Skipped 5 previous similar messages [14089.931204] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure == 18:29:43 (1476494983) [14090.025369] Lustre: lustre-OST0000: Client d158399d-7705-3049-2b52-0cf9ec22ab30 (at 10.2.4.36@tcp) reconnecting [14091.021583] LustreError: 49370:0:(ldlm_lib.c:3238:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8803f5b45050 x1548215508310068/t0(0) o3->d158399d-7705-3049-2b52-0cf9ec22ab30@10.2.4.36@tcp:-1/-1 lens 608/432 e 0 to 0 dl 1476495026 ref 1 fl Interpret:/0/0 rc 0/0 [14091.057555] Lustre: lustre-OST0000: Bulk IO read error with d158399d-7705-3049-2b52-0cf9ec22ab30 (at 10.2.4.36@tcp), client will retry: rc -110 [14092.208486] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC == 18:29:45 (1476494985) [14128.380173] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe == 18:30:21 (1476495021) [14141.853719] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 == 18:30:35 (1476495035) [14155.207620] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type == 18:30:48 (1476495048) [14156.516226] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir == 18:30:50 (1476495050) [14156.962091] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [14158.127945] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM == 18:30:51 (1476495051) [14159.362902] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks == 18:30:52 (1476495052) [14214.396233] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart == 18:31:47 (1476495107) [14274.865461] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [14274.865486] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [14274.866317] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff3e8f3b to 0x20295361ff59f971 [14274.921155] LustreError: Skipped 1 previous similar message [14274.930663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [14274.955684] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [14277.243507] Lustre: lustre-OST0000: deleting orphan objects from 0x0:177182 to 0x0:178753 [14289.072941] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf == 18:33:02 (1476495182) [14380.513067] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files == 18:34:33 (1476495273) [14381.232519] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [14382.402683] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory == 18:34:35 (1476495275) [14382.791145] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [14383.972675] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory == 18:34:37 (1476495277) [14384.378695] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [14385.643291] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed == 18:34:38 (1476495278) [14386.041571] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [14387.225391] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory == 18:34:40 (1476495280) [14387.635093] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [14388.823168] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files == 18:34:42 (1476495282) [14389.208500] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [14390.372251] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 18:34:43 (1476495283) [14390.764508] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [14391.949038] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT == 18:34:45 (1476495285) [14392.330100] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [14393.508134] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root == 18:34:46 (1476495286) [14393.891157] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [14395.061681] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes == 18:34:48 (1476495288) [14395.447989] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [14396.635982] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC == 18:34:49 (1476495289) [14399.778017] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer == 18:34:53 (1476495293) [14407.653478] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount == 18:35:00 (1476495300) [14408.044611] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [14408.059441] Lustre: Skipped 4 previous similar messages [14409.236183] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds == 18:35:02 (1476495302) [14410.805733] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds == 18:35:04 (1476495304) [14412.335912] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM == 18:35:05 (1476495305) [14413.060368] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.44 too old [14414.279396] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly == 18:35:07 (1476495307) [14417.886862] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file == 18:35:11 (1476495311) [14419.292622] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls == 18:35:12 (1476495312) [14420.530705] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency == 18:35:13 (1476495313) [14422.215902] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test == 18:35:15 (1476495315) [14449.235760] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly == 18:35:42 (1476495342) [14455.612314] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly == 18:35:48 (1476495348) [14460.519738] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) == 18:35:53 (1476495353) [14460.911636] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [14462.075954] Lustre: DEBUG MARKER: == sanity test 241: bio vs dio == 18:35:55 (1476495355) [14477.517265] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio == 18:36:10 (1476495370) [14485.440643] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable == 18:36:18 (1476495378) [14487.298348] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests == 18:36:20 (1476495380) [14516.400598] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests == 18:36:49 (1476495409) [14534.633649] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs == 18:37:07 (1476495427) [14535.894249] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length == 18:37:09 (1476495429) [14537.075676] Lustre: *** cfs_fail_loc=234, val=0*** [14537.084109] Lustre: Skipped 24 previous similar messages [14538.300517] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size == 18:37:11 (1476495431) [14539.495936] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit == 18:37:12 (1476495432) [14540.743699] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly == 18:37:13 (1476495433) [14542.051493] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool == 18:37:15 (1476495435) [14544.687343] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state == 18:37:17 (1476495437) [14549.998730] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [14550.013599] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [14557.001352] Lustre: 23245:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476495443/real 1476495443] req@ffff88041a8da100 x1548202901288640/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476495450 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [14557.040603] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [14557.062609] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff59f971 to 0x20295361ff810953 [14565.961830] Lustre: lustre-OST0000: deleting orphan objects from 0x0:207794 to 0x0:207873 [14568.283336] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test == 18:37:41 (1476495461) [14568.658686] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [14569.891392] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir == 18:37:43 (1476495463) [14570.291105] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [14571.475818] Lustre: DEBUG MARKER: == sanity test 300c: chown [14571.857969] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [14573.031292] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory == 18:37:46 (1476495466) [14573.434252] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [14574.585470] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory == 18:37:47 (1476495467) [14575.304923] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [14576.492144] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory == 18:37:49 (1476495469) [14577.231255] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [14578.413433] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory == 18:37:51 (1476495471) [14579.155709] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [14580.316299] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory == 18:37:53 (1476495473) [14581.041579] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [14582.074430] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [14582.221275] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory == 18:37:55 (1476495475) [14582.944579] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [14584.112823] Lustre: DEBUG MARKER: == sanity test 300j: test large update record == 18:37:57 (1476495477) [14584.842249] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [14586.017407] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory == 18:37:59 (1476495479) [14586.718830] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [14587.893895] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout == 18:38:01 (1476495481) [14588.635240] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [14589.807588] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS == 18:38:03 (1476495483) [14591.392367] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA == 18:38:04 (1476495484) [14592.119636] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [14593.304784] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [14593.703042] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space == 18:38:06 (1476495486) [14594.096066] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [14595.278406] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory == 18:38:08 (1476495488) [14595.657152] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [14596.841432] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file == 18:38:10 (1476495490) [14597.233341] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [14598.414565] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open == 18:38:11 (1476495491) [14598.828593] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [14600.009134] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links == 18:38:13 (1476495493) [14600.412448] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [14601.584117] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link == 18:38:14 (1476495494) [14602.972900] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled == 18:38:16 (1476495496) [14604.189209] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively == 18:38:17 (1476495497) [14605.431671] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error == 18:38:18 (1476495498) [14606.609404] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea == 18:38:19 (1476495499) [14608.469353] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread == 18:38:21 (1476495501) [14629.291251] Lustre: 75495:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476495516/real 1476495516] req@ffff8803f51e0000 x1548202901289008/t0(0) o39->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 224/224 e 0 to 1 dl 1476495522 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [14636.020898] Lustre: server umount lustre-OST0000 complete [14659.867287] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [14659.895253] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [14662.503500] Lustre: lustre-OST0000: deleting orphan objects from 0x0:207884 to 0x0:207905 [14690.070984] Lustre: DEBUG MARKER: Using TIMEOUT=100 [14696.915577] Lustre: DEBUG MARKER: == sanity test complete, duration 5520 sec == 18:39:50 (1476495590) [17286.619926] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [17286.634995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [17300.710632] Lustre: 76410:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476498186/real 1476498186] req@ffff8803f51e0000 x1548202901410096/t0(0) o251->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476498192 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [17300.750970] Lustre: 76410:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [17301.121188] Lustre: server umount lustre-OST0000 complete [17316.578031] attempt to access beyond end of device [17316.586699] sdb1: rw=0, want=4208970, limit=4208969 [17316.608587] attempt to access beyond end of device [17316.617076] sdb1: rw=0, want=4208970, limit=4208969 [17316.627465] attempt to access beyond end of device [17316.635924] sdb1: rw=0, want=4208970, limit=4208969 [17316.693418] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17316.781849] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17332.354644] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17332.368307] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17332.873220] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17332.931254] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [17333.325201] Lustre: lustre-OST0000: new disk, initializing [17333.334689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space [17334.156319] attempt to access beyond end of device [17334.164765] sdb1: rw=0, want=4208976, limit=4208969 [17338.837298] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [17338.849692] Lustre: Skipped 6 previous similar messages [17352.277702] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17366.006269] Lustre: DEBUG MARKER: Client: Lustre version: 2.8.59 [17366.705452] Lustre: DEBUG MARKER: MDS: Lustre version: 2.8.59 [17367.411762] Lustre: DEBUG MARKER: OSS: Lustre version: 2.8.59 [17367.807853] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 19:24:19 PDT 2016 [17371.563937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17374.279234] Lustre: DEBUG MARKER: excepting tests: 76 101g 42a 42b 42c 42d 45 68b [17374.680739] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o [17375.171759] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 19:24:27 (1476498267) [17376.788460] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 19:24:28 (1476498268) [17378.352159] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 19:24:30 (1476498270) [17380.394513] Lustre: DEBUG MARKER: == sanity test complete, duration 12 sec ============================================================= 19:24:32 (1476498272) [17383.380548] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [17383.394992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [17394.980021] Lustre: lustre-OST0000: Not available for connect from 10.2.4.37@tcp (stopping) [17395.385568] Lustre: 23237:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476498280/real 1476498280] req@ffff8803ec106600 x1548202901410992/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476498287 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17395.424893] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [17401.350201] Lustre: server umount lustre-OST0000 complete [17477.214432] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17477.238337] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,,errors=remount-ro,no_mbcache,nodelalloc [17485.907765] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [17485.920024] Lustre: Skipped 2 previous similar messages [17485.979526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4 to 0x0:33 [17531.885135] Lustre: DEBUG MARKER: Client: Lustre version: 2.8.59 [17532.564538] Lustre: DEBUG MARKER: MDS: Lustre version: 2.8.59 [17533.251527] Lustre: DEBUG MARKER: OSS: Lustre version: 2.8.59 [17533.650477] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 19:27:05 PDT 2016 [17537.388046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17539.894376] Lustre: DEBUG MARKER: excepting tests: 76 101g 42a 42b 42c 42d 45 68b [17540.278217] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o [17540.712177] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 19:27:12 (1476498432) [17542.298176] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 19:27:14 (1476498434) [17543.901334] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 19:27:15 (1476498435) [17545.981069] Lustre: DEBUG MARKER: == sanity test complete, duration 12 sec ============================================================= 19:27:17 (1476498437) [17634.072821] Lustre: DEBUG MARKER: Client: Lustre version: 2.8.59 [17634.777072] Lustre: DEBUG MARKER: MDS: Lustre version: 2.8.59 [17635.474087] Lustre: DEBUG MARKER: OSS: Lustre version: 2.8.59 [17635.859749] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 19:28:47 PDT 2016 [17639.643012] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17642.175429] Lustre: DEBUG MARKER: excepting tests: 76 101g 42a 42b 42c 42d 45 68b [17642.557435] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o [17643.014675] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 19:28:54 (1476498534) [17644.593960] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 19:28:56 (1476498536) [17646.169663] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 19:28:57 (1476498537) [17647.758771] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 19:28:59 (1476498539) [17649.446836] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 19:29:01 (1476498541) [17651.041322] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 19:29:02 (1476498542) [17652.723115] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 19:29:04 (1476498544) [17654.331075] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 19:29:06 (1476498546) [17656.069722] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 19:29:07 (1476498547) [17657.669599] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 19:29:09 (1476498549) [17659.276290] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ========= 19:29:11 (1476498551) [17660.884911] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 19:29:12 (1476498552) [17662.598121] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) ================ 19:29:14 (1476498554) [17664.266111] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 19:29:16 (1476498556) [17665.857580] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 19:29:17 (1476498557) [17667.449901] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 19:29:19 (1476498559) [17669.053533] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 19:29:20 (1476498560) [17670.647276] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 19:29:22 (1476498562) [17672.232169] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 19:29:23 (1476498563) [17673.964489] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 19:29:25 (1476498565) [17675.571272] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 19:29:27 (1476498567) [17677.205591] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 19:29:28 (1476498568) [17678.805794] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 19:29:30 (1476498570) [17680.410225] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 19:29:32 (1476498572) [17682.032049] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 19:29:33 (1476498573) [17683.663309] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 19:29:35 (1476498575) [17685.253941] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 19:29:36 (1476498576) [17686.855145] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 19:29:38 (1476498578) [17688.462554] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 19:29:40 (1476498580) [17690.068539] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 19:29:41 (1476498581) [17691.744115] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 19:29:43 (1476498583) [17694.558153] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 19:29:46 (1476498586) [17696.502436] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 19:29:48 (1476498588) [17698.471145] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 19:29:50 (1476498590) [17700.176015] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 19:29:51 (1476498591) [17701.765416] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 19:29:53 (1476498593) [17717.702577] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [17717.718442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [17724.705254] Lustre: 23251:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476498609/real 1476498609] req@ffff8803f5640c00 x1548202901413008/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476498616 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17724.747955] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [17725.337579] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [17725.350902] Lustre: Skipped 3 previous similar messages [17730.553041] Lustre: lustre-OST0000: deleting orphan objects from 0x0:59 to 0x0:97 [17732.089529] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 19:30:23 (1476498623) [17733.159162] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [17734.714559] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 19:30:26 (1476498626) [17734.776138] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff86ac79 to 0x20295361ff888084 [17746.780972] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [17746.803698] LustreError: 78816:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [17762.566014] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [17763.421696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [17763.811188] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476498644/real 1476498644] req@ffff8803f5a2c500 x1548202901413216/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476498655 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17763.853182] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [17766.826654] LustreError: 78816:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [17766.839969] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ff888084 to 0x20295361ff88831d [17766.860599] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [17766.973573] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 19:30:58 (1476498658) [17768.663800] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 19:31:00 (1476498660) [17770.496212] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 19:31:02 (1476498662) [17772.117484] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 19:31:03 (1476498663) [17773.774911] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 19:31:05 (1476498665) [17775.328073] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 19:31:07 (1476498667) [17775.705844] Lustre: DEBUG MARKER: 1 done [17776.147572] Lustre: DEBUG MARKER: 2 done [17776.549344] Lustre: DEBUG MARKER: 3 done [17778.107481] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 19:31:09 (1476498669) [17779.737960] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 19:31:11 (1476498671) [17781.562016] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 19:31:13 (1476498673) [17783.173585] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 19:31:14 (1476498674) [17784.791466] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 19:31:16 (1476498676) [17786.414377] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 19:31:18 (1476498678) [17788.042152] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 19:31:19 (1476498679) [17789.669759] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 19:31:21 (1476498681) [17791.263723] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 19:31:22 (1476498682) [17792.893412] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 19:31:24 (1476498684) [17794.483101] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 19:31:26 (1476498686) [17796.103292] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 19:31:27 (1476498687) [17797.747737] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 19:31:29 (1476498689) [17799.362985] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 19:31:31 (1476498691) [17800.982374] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 19:31:32 (1476498692) [17802.572192] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 19:31:34 (1476498694) [17804.170897] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 19:31:35 (1476498695) [17805.805372] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 19:31:37 (1476498697) [17807.411894] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 19:31:39 (1476498699) [17889.528273] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 19:33:01 (1476498781) [17891.186161] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 19:33:02 (1476498782) [17892.862878] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 19:33:04 (1476498784) [17894.461731] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 19:33:06 (1476498786) [17896.081600] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 19:33:07 (1476498787) [17897.710077] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 19:33:09 (1476498789) [17899.419564] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 19:33:11 (1476498791) [17899.808523] Lustre: DEBUG MARKER: SKIP: sanity test_24v not enough free inodes 98608 required 100000 [17901.416241] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 19:33:13 (1476498793) [17903.081297] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 19:33:14 (1476498794) [17903.482242] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [17905.043923] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 19:33:16 (1476498796) [17905.464054] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [17907.103829] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 19:33:18 (1476498798) [17923.220749] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 19:33:34 (1476498814) [17923.609716] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [17925.180556] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 19:33:36 (1476498816) [17925.555997] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [17927.165825] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [17927.574454] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 19:33:39 (1476498819) [17927.986644] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [17929.606349] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 19:33:41 (1476498821) [17931.341582] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 19:33:42 (1476498822) [17932.919485] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 19:33:44 (1476498824) [17934.527552] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 19:33:46 (1476498826) [17936.124010] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 19:33:47 (1476498827) [17937.759010] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 19:33:49 (1476498829) [17939.463518] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 19:33:51 (1476498831) [17941.013490] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 19:33:52 (1476498832) [17942.665659] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 19:33:54 (1476498834) [17943.076354] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [17944.673314] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 19:33:56 (1476498836) [17945.059249] Lustre: DEBUG MARKER: SKIP: sanity test_27b skipping 2-stripe test [17946.626792] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 19:33:58 (1476498838) [17948.234165] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 19:33:59 (1476498839) [17950.191767] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 19:34:01 (1476498841) [17951.818711] Lustre: DEBUG MARKER: == sanity test 27g: /bin/lfs getstripe with no objects =============================================== 19:34:03 (1476498843) [17953.647252] Lustre: DEBUG MARKER: == sanity test 27i: /bin/lfs getstripe with some objects ============================================= 19:34:05 (1476498845) [17955.312919] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 19:34:06 (1476498846) [17956.946728] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 19:34:08 (1476498848) [17958.934264] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 19:34:10 (1476498850) [17961.049857] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [17961.443541] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 19:34:13 (1476498853) [17961.829688] Lustre: DEBUG MARKER: SKIP: sanity test_27n too few OSTs [17963.410152] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 19:34:15 (1476498855) [17963.805207] Lustre: DEBUG MARKER: SKIP: sanity test_27o too few OSTs [17965.380057] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 19:34:16 (1476498856) [17965.776700] Lustre: DEBUG MARKER: SKIP: sanity test_27p too few OSTs [17967.344117] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 19:34:18 (1476498858) [17967.738215] Lustre: DEBUG MARKER: SKIP: sanity test_27q too few OSTs [17969.298330] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 19:34:20 (1476498860) [17969.691768] Lustre: DEBUG MARKER: SKIP: sanity test_27r too few OSTs [17971.332206] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 19:34:22 (1476498862) [17972.914878] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 19:34:24 (1476498864) [17974.509450] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 19:34:26 (1476498866) [17974.907780] Lustre: DEBUG MARKER: SKIP: sanity test_27u too few OSTs [17976.586862] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 19:34:28 (1476498868) [17976.977001] Lustre: DEBUG MARKER: SKIP: sanity test_27v too few OSTs [17978.562841] Lustre: DEBUG MARKER: == sanity test 27w: check /bin/lfs setstripe -S option =============================================== 19:34:30 (1476498870) [17980.143346] Lustre: DEBUG MARKER: == sanity test 27wa: check /bin/lfs setstripe -c -i options ========================================== 19:34:31 (1476498871) [17980.587161] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [17982.177366] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 19:34:33 (1476498873) [17982.599136] Lustre: DEBUG MARKER: SKIP: sanity test_27x 1 < 2 OSTs [17984.184554] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 19:34:35 (1476498875) [17984.589865] Lustre: DEBUG MARKER: SKIP: sanity test_27y 1 < 2 OSTs -- skipping [17986.225472] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 19:34:37 (1476498877) [17991.889627] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [17992.305227] Lustre: DEBUG MARKER: FID seq 0x200000bd1, oid 0x4171 ver 0x0 [17992.697529] Lustre: DEBUG MARKER: LOV seq 0x200000bd1, oid 0x4171, count: 1 [17993.104677] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:165/0xa5 seq:0 [17993.770737] Lustre: Failing over lustre-OST0000 [17994.297445] Lustre: server umount lustre-OST0000 complete [17995.383295] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17995.426469] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [17997.657025] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [17997.684097] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [18001.133630] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18001.147316] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [18001.161929] Lustre: Skipped 4 previous similar messages [18005.619390] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [18005.621203] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [18006.133637] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [18006.555177] Lustre: DEBUG MARKER: FID seq 0x200000bd1, oid 0x4173 ver 0x0 [18006.939458] Lustre: DEBUG MARKER: LOV seq 0x200000bd1, oid 0x4173, count: 1 [18007.343066] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:166/0xa6 seq:0 [18008.000931] Lustre: Failing over lustre-OST0000 [18008.695276] Lustre: server umount lustre-OST0000 complete [18009.728997] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [18009.772032] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [18012.151563] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [18012.166925] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [18017.727451] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18022.066297] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [18022.068189] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:225 [18023.758560] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 19:35:15 (1476498915) [18025.411069] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 19:35:17 (1476498917) [18027.073927] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 19:35:18 (1476498918) [18027.463271] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [18029.065117] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 19:35:20 (1476498920) [18029.473683] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [18031.128700] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 19:35:22 (1476498922) [18031.541559] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [18033.143259] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 19:35:24 (1476498924) [18034.196348] Lustre: Failing over lustre-OST0000 [18034.914448] Lustre: server umount lustre-OST0000 complete [18036.838762] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [18036.863166] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [18037.145694] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18037.498024] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18038.991407] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18038.993356] Lustre: lustre-OST0000: deleting orphan objects from 0x0:228 to 0x0:257 [18047.744346] Lustre: DEBUG MARKER: sanity test_27F: @@@@@@ FAIL: wrong stripecount [18051.791030] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 19:35:43 (1476498943) [18053.542102] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 19:35:45 (1476498945) [18055.036575] Lustre: DEBUG MARKER: first d29 [18055.414846] Lustre: DEBUG MARKER: second d29 [18055.806097] Lustre: DEBUG MARKER: done [18057.335931] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 19:35:48 (1476498948) [18058.930675] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 19:35:50 (1476498950) [18060.540571] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 19:35:52 (1476498952) [18062.275155] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 19:35:53 (1476498953) [18063.836909] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 19:35:55 (1476498955) [18065.473630] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 19:35:57 (1476498957) [18067.103651] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 19:35:58 (1476498958) [18068.723913] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 19:36:00 (1476498960) [18070.307953] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 19:36:01 (1476498961) [18078.065863] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 19:36:09 (1476498969) [18079.737881] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 19:36:11 (1476498971) [18081.372125] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 19:36:12 (1476498972) [18083.043558] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 19:36:14 (1476498974) [18084.654722] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 19:36:16 (1476498976) [18086.338077] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 19:36:17 (1476498977) [18087.974858] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 19:36:19 (1476498979) [18089.602272] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 19:36:21 (1476498981) [18099.084197] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 19:36:30 (1476498990) [18099.501073] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [18101.095529] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 19:36:32 (1476498992) [18102.742853] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 19:36:34 (1476498994) [18104.448618] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 19:36:35 (1476498995) [18106.093580] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 19:36:37 (1476498997) [18107.761500] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 19:36:39 (1476498999) [18109.367045] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 19:36:40 (1476499000) [18111.130798] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 19:36:42 (1476499002) [18112.916316] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 19:36:44 (1476499004) [18114.667468] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 19:36:46 (1476499006) [18116.384618] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 19:36:47 (1476499007) [18118.229118] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 19:36:49 (1476499009) [18119.935319] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 19:36:51 (1476499011) [18121.680965] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 19:36:53 (1476499013) [18123.497470] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 19:36:54 (1476499014) [18125.195308] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 19:36:56 (1476499016) [18127.461044] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 19:36:58 (1476499018) [18127.842544] Lustre: DEBUG MARKER: 32p_1 [18128.289665] Lustre: DEBUG MARKER: 32p_2 [18128.743023] Lustre: DEBUG MARKER: 32p_3 [18129.164468] Lustre: DEBUG MARKER: 32p_4 [18129.561274] Lustre: DEBUG MARKER: 32p_5 [18129.932428] Lustre: DEBUG MARKER: 32p_6 [18130.430181] Lustre: DEBUG MARKER: 32p_7 [18130.827391] Lustre: DEBUG MARKER: 32p_8 [18131.215779] Lustre: DEBUG MARKER: 32p_9 [18131.661594] Lustre: DEBUG MARKER: 32p_10 [18133.306665] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 19:37:04 (1476499024) [18135.004217] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 19:37:06 (1476499026) [18136.747655] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 19:37:08 (1476499028) [18137.164583] Lustre: DEBUG MARKER: 33_1 [18137.530828] Lustre: DEBUG MARKER: 33_2 [18139.374739] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 19:37:10 (1476499030) [18141.137144] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 19:37:12 (1476499032) [18142.920260] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 19:37:14 (1476499034) [18144.926231] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 19:37:16 (1476499036) [18145.328587] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [18146.996820] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 19:37:18 (1476499038) [18147.399036] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [18149.665631] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 19:37:21 (1476499041) [18150.094909] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [18151.756777] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 19:37:23 (1476499043) [18153.452286] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 19:37:24 (1476499044) [18155.190093] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 19:37:26 (1476499046) [18156.876947] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 19:37:28 (1476499048) [18158.597876] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 19:37:30 (1476499050) [18160.316794] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 19:37:31 (1476499051) [18162.107558] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 19:37:33 (1476499053) [18164.038892] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 19:37:35 (1476499055) [18165.963615] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 19:37:37 (1476499057) [18170.408908] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 19:37:41 (1476499061) [18172.204419] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 19:37:43 (1476499063) [18173.928703] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 19:37:45 (1476499065) [18175.638322] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 19:37:47 (1476499067) [18177.488307] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 19:37:48 (1476499068) [18179.546356] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 19:37:51 (1476499071) [18181.236246] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 19:37:52 (1476499072) [18184.007341] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 19:37:55 (1476499075) [18228.698929] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 19:38:40 (1476499120) [18231.412302] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 19:38:42 (1476499122) [18231.832651] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [18233.509325] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 19:38:44 (1476499124) [18235.173185] Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create =========================================================== 19:38:46 (1476499126) [18238.886671] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 19:38:50 (1476499130) [18241.800450] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 19:38:53 (1476499133) [18245.552828] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 19:38:57 (1476499137) [18247.230810] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 19:38:58 (1476499138) [18248.918175] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 19:39:00 (1476499140) [18252.566822] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 19:39:04 (1476499144) [18256.278286] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 19:39:07 (1476499147) [18258.989647] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 19:39:10 (1476499150) [18261.705074] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 19:39:13 (1476499153) [18265.363759] Lustre: 2 MB is too small for debug buffer size, setting it to 128 MB. [18267.528703] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 19:39:18 (1476499158) [18270.522145] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 19:39:21 (1476499161) [18279.807567] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 19:39:31 (1476499171) [18283.497464] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 19:39:34 (1476499174) [18296.372909] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 19:39:47 (1476499187) [18298.033879] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 19:39:49 (1476499189) [18298.444529] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [18300.037765] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 19:39:51 (1476499191) [18301.708217] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 19:39:53 (1476499193) [18303.285660] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 19:39:54 (1476499194) [18305.955123] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [18306.376565] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [18306.794643] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [18307.210792] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [18307.619268] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 19:39:59 (1476499199) [18327.470538] Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY ========================= 19:40:18 (1476499218) [18330.482077] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 19:40:21 (1476499221) [18332.252264] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 19:40:23 (1476499223) [18333.952116] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre======================================================== 19:40:25 (1476499225) [18335.643778] Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe ============================================= 19:40:27 (1476499227) [18336.053433] Lustre: DEBUG MARKER: SKIP: sanity test_44 skipping 2-stripe test [18337.690634] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 19:40:29 (1476499229) [18339.586557] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [18340.018853] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 19:40:31 (1476499231) [18341.822273] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 19:40:33 (1476499233) [18343.818424] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 19:40:35 (1476499235) [18345.620259] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 19:40:37 (1476499237) [18347.358382] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 19:40:38 (1476499238) [18349.124889] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 19:40:40 (1476499240) [18350.893133] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 19:40:42 (1476499242) [18354.899473] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 19:40:46 (1476499246) [18356.535992] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 19:40:47 (1476499247) [18358.525817] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ===================== 19:40:49 (1476499249) [18358.967319] Lustre: DEBUG MARKER: SKIP: sanity test_51b not enough blocks (27781) [18360.581147] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 19:40:51 (1476499251) [18360.984032] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping test with few OSTs [18362.578108] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 19:40:53 (1476499253) [18775.927633] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 19:47:47 (1476499667) [18777.667683] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 19:47:48 (1476499668) [18779.370992] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 19:47:50 (1476499670) [18783.667673] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 19:47:54 (1476499674) [18785.551470] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 19:47:56 (1476499676) [18787.197864] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 19:47:58 (1476499678) [18789.964558] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 19:48:01 (1476499681) [18791.578366] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 19:48:02 (1476499682) [18793.222850] Lustre: DEBUG MARKER: == sanity test 56a: check /bin/lfs getstripe ========================================================= 19:48:04 (1476499684) [18793.955815] Lustre: DEBUG MARKER: SKIP: sanity test_56a skipping other /bin/lfs getstripe --obd test [18795.649875] Lustre: DEBUG MARKER: == sanity test 56b: check /bin/lfs getdirstripe ====================================================== 19:48:06 (1476499686) [18797.333519] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 19:48:08 (1476499688) [18799.201687] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 19:48:10 (1476499690) [18801.035201] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 19:48:12 (1476499692) [18802.682914] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 19:48:13 (1476499693) [18804.543923] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 19:48:15 (1476499695) [18806.348219] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 19:48:17 (1476499697) [18808.195018] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 19:48:19 (1476499699) [18810.139560] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 19:48:21 (1476499701) [18811.982810] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 19:48:23 (1476499703) [18813.989894] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 19:48:25 (1476499705) [18816.128299] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 19:48:27 (1476499707) [18818.060127] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 19:48:29 (1476499709) [18820.190225] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 19:48:31 (1476499711) [18822.128544] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 19:48:33 (1476499713) [18824.183714] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 19:48:35 (1476499715) [18826.222137] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 19:48:37 (1476499717) [18828.244445] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 19:48:39 (1476499719) [18828.648988] Lustre: DEBUG MARKER: SKIP: sanity test_56w 1 < 2 OSTs -- skipping [18830.285789] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 19:48:41 (1476499721) [18830.730372] Lustre: DEBUG MARKER: SKIP: sanity test_56x need 2 OST, skipping test [18832.337764] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 19:48:43 (1476499723) [18832.750424] Lustre: DEBUG MARKER: SKIP: sanity test_56xa need 2 OST, skipping test [18834.347112] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 19:48:45 (1476499725) [18836.412719] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 19:48:47 (1476499727) [18838.450360] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 19:48:49 (1476499729) [18838.857669] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [18840.512755] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 19:48:51 (1476499731) [18843.107736] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 19:48:54 (1476499734) [18848.355100] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 19:48:59 (1476499739) [18849.955542] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 19:49:01 (1476499741) [18858.728837] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 19:49:09 (1476499749) [18859.471352] Lustre: DEBUG MARKER: test_60 run 5325 - from kernel mode [18882.700537] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [18882.716177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [18889.703176] Lustre: 23241:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476499774/real 1476499774] req@ffff8803f5644200 x1548202901435296/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476499781 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [18889.745122] Lustre: 23241:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18889.759680] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [18897.539117] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [18897.552215] Lustre: Skipped 8 previous similar messages [18899.786991] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295361ffa2e720 to 0x2029536200087486 [18900.806518] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476499781/real 1476499781] req@ffff8803f5643000 x1548202901435376/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476499792 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [18900.849140] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18901.291993] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 19:49:52 (1476499792) [18904.789264] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [18905.283703] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2100 to 0x0:2145 [18906.899088] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 19:49:58 (1476499798) [18931.977808] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 19:50:23 (1476499823) [18933.604044] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 19:50:24 (1476499824) [18935.610696] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 19:50:26 (1476499826) [18937.385110] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 19:50:28 (1476499828) [18999.276503] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 19:51:30 (1476499890) [19004.250075] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 19:51:35 (1476499895) [19005.889286] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [19006.313236] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ =============================== 19:51:37 (1476499897) [19007.901772] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 19:51:38 (1476499898) [19009.607333] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 19:51:40 (1476499900) [19011.287808] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 =================================== 19:51:42 (1476499902) [19012.917363] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 19:51:44 (1476499904) [19014.627506] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 19:51:45 (1476499905) [19016.325141] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 19:51:47 (1476499907) [19018.185227] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 19:51:49 (1476499909) [19019.892814] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 19:51:50 (1476499910) [19021.615959] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 19:51:52 (1476499912) [19023.299200] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 19:51:54 (1476499914) [19026.060609] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 19:51:57 (1476499917) [19028.542989] Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping ======================================= 19:51:59 (1476499919) [19034.688926] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 19:52:05 (1476499925) [19037.390407] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 19:52:08 (1476499928) [19037.798333] Lustre: DEBUG MARKER: SKIP: sanity test_65k too few OSTs [19039.459572] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 19:52:10 (1476499930) [19041.176938] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 19:52:12 (1476499932) [19042.798805] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 19:52:13 (1476499933) [19046.297959] Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) [19046.711544] Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b [19047.114610] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 19:52:18 (1476499938) [19048.129121] Lustre: *** cfs_fail_loc=217, val=0*** [19049.968436] Lustre: *** cfs_fail_loc=217, val=0*** [19049.968439] Lustre: *** cfs_fail_loc=217, val=0*** [19049.968444] Lustre: Skipped 7 previous similar messages [19049.995644] Lustre: Skipped 1 previous similar message [19051.941330] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [19052.356097] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 19:52:23 (1476499943) [19054.346689] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 19:52:25 (1476499945) [19056.554329] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 19:52:27 (1476499947) [19084.203017] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 19:52:55 (1476499975) [19086.011466] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 19:52:57 (1476499977) [19087.635015] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 19:52:58 (1476499978) [19089.241329] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [19089.640962] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 19:53:00 (1476499980) [19092.164921] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 19:53:03 (1476499983) [19092.260967] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e99:0x0] object 0x0:7171 extent [0-1048575]: client csum ea2d05b3, server csum ea2d05b2 [19093.943021] Lustre: DEBUG MARKER: set checksum type to crc32 [19095.639536] Lustre: DEBUG MARKER: set checksum type to adler [19097.283374] Lustre: DEBUG MARKER: set checksum type to adler [19098.862262] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 19:53:09 (1476499989) [19099.008278] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e9a:0x0] object 0x0:7172 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [19103.661884] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 19:53:14 (1476499994) [19104.081295] Lustre: DEBUG MARKER: set checksum type to crc32 [19104.218624] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e9c:0x0] object 0x0:7173 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [19104.246038] LustreError: Skipped 1 previous similar message [19107.403726] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e9c:0x0] object 0x0:7173 extent [5242880-6291455]: client csum b2f1b12, server csum b2f1b11 [19107.430445] LustreError: Skipped 14 previous similar messages [19114.678977] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e9c:0x0] object 0x0:7173 extent [3145728-4194303]: client csum b2f1b12, server csum b2f1b11 [19114.705629] LustreError: Skipped 15 previous similar messages [19116.804033] Lustre: DEBUG MARKER: set checksum type to adler [19124.548542] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200000bd1:0x5e9c:0x0] object 0x0:7173 extent [4194304-5242879]: client csum 19eeae62, server csum 19eeae61 [19124.575703] LustreError: Skipped 32 previous similar messages [19129.954517] Lustre: DEBUG MARKER: set checksum type to adler [19131.578914] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 19:53:42 (1476500022) [19132.031667] Lustre: *** cfs_fail_loc=21a, val=0*** [19134.609771] Lustre: *** cfs_fail_loc=21b, val=0*** [19137.998437] Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM ========================================== 19:53:49 (1476500029) [19140.863961] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 19:53:51 (1476500031) [19144.839731] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 19:53:55 (1476500035) [19149.011170] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 19:54:00 (1476500040) [19159.381521] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 19:54:10 (1476500050) [19162.480602] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 19:54:13 (1476500053) [19162.857420] Lustre: *** cfs_fail_loc=228, val=0*** [19162.866116] Lustre: Skipped 3 previous similar messages [19164.904383] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 19:54:15 (1476500055) [19165.241654] Lustre: *** cfs_fail_loc=228, val=0*** [19167.674423] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 19:54:18 (1476500058) [19171.885385] Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 19:54:22 (1476500062) [19174.080274] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 19:54:25 (1476500065) [19175.858541] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 19:54:26 (1476500066) [19178.229758] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 19:54:29 (1476500069) [19179.907320] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 19:54:30 (1476500070) [19181.624888] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 19:54:32 (1476500072) [19185.324791] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 19:54:36 (1476500076) [19187.143871] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 19:54:38 (1476500078) [19205.340098] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 19:54:56 (1476500096) [19205.743858] Lustre: DEBUG MARKER: SKIP: sanity test_101b skipping stride IO stride-ahead test [19207.313377] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 19:54:58 (1476500098) [19231.338663] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 19:55:22 (1476500122) [19231.757953] Lustre: DEBUG MARKER: SKIP: sanity test_101d Need free space 500M, have 153M [19233.362974] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 19:55:24 (1476500124) [19241.371858] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 19:55:32 (1476500132) [19248.400574] Lustre: DEBUG MARKER: SKIP: sanity test_101g skipping ALWAYS excluded test 101g [19248.821766] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 19:55:39 (1476500139) [19249.240630] Lustre: DEBUG MARKER: sanity test_102a: @@@@@@ FAIL: setfattr -n trusted.name1=value1 /mnt/lustre/f102a.sanity failed [19251.700994] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 19:55:42 (1476500142) [19252.111247] Lustre: DEBUG MARKER: SKIP: sanity test_102b skipping 2-stripe test [19253.719844] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 19:55:44 (1476500144) [19254.165095] Lustre: DEBUG MARKER: SKIP: sanity test_102c skipping 2-stripe test [19255.758910] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 19:55:46 (1476500146) [19256.218282] Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed [19257.839817] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 19:55:48 (1476500148) [19258.246714] Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed [19259.870961] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 19:55:50 (1476500150) [19260.283609] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [19260.712605] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [19261.118706] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [19261.502235] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [19263.127002] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 19:55:54 (1476500154) [19263.901785] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [19265.508964] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 19:55:56 (1476500156) [19267.137690] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 19:55:58 (1476500158) [19267.548958] Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed [19269.195242] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 19:56:00 (1476500160) [19270.875169] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 19:56:01 (1476500161) [19272.490347] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 19:56:03 (1476500163) [19274.130783] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 19:56:05 (1476500165) [19276.543316] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 19:56:07 (1476500167) [19278.585962] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 19:56:09 (1476500169) [19280.515113] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 19:56:11 (1476500171) [19282.526045] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 19:56:13 (1476500173) [19295.802173] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' ======================================================== 19:56:26 (1476500186) [19299.980984] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [19299.996365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [19306.983554] Lustre: 23242:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476500191/real 1476500191] req@ffff8803f5645100 x1548202901455344/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476500198 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19307.024771] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [19307.047676] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200087486 to 0x202953620011f919 [19307.068068] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19310.523323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8394 to 0x0:8417 [19312.049702] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [19312.065169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [19319.052413] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [19319.074703] LustreError: 102138:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [19324.648488] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 19:56:55 (1476500215) [19325.078416] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620011f919 to 0x202953620011fb0a [19325.479117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8419 to 0x0:8449 [19327.343764] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 19:56:58 (1476500218) [19327.469980] Lustre: lustre-OST0000: Client 6ee6bf77-2716-7d2b-5f3a-f5ce73284f1a (at 10.2.4.36@tcp) reconnecting [19327.933120] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880036dd0000.ost_server_uuid in FULL state after 0 sec [19329.607710] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 19:57:00 (1476500220) [19331.211050] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 19:57:02 (1476500222) [19332.955567] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 19:57:03 (1476500223) [19334.647121] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 19:57:05 (1476500225) [19335.083148] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19336.314359] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 19:57:07 (1476500227) [19348.026942] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 19:57:18 (1476500238) [19349.777850] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 19:57:20 (1476500240) [19351.462619] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 19:57:22 (1476500242) [19354.157901] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 19:57:25 (1476500245) [19355.883969] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [19356.294489] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 19:57:27 (1476500247) [19356.725869] Lustre: DEBUG MARKER: SKIP: sanity test_116a 1 < 2 OSTs [19358.346896] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 19:57:29 (1476500249) [19361.721380] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 19:57:32 (1476500252) [19363.424069] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 19:57:34 (1476500254) [19365.379876] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 19:57:36 (1476500256) [19365.860186] Lustre: *** cfs_fail_loc=217, val=0*** [19368.349538] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 19:57:39 (1476500259) [19368.851644] LustreError: 102151:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 sleeping for 1000ms [19369.867001] LustreError: 102151:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 awake [19373.454205] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 19:57:44 (1476500264) [19374.026880] LustreError: 102151:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 5000ms [19379.044448] LustreError: 102151:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [19381.332642] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 19:57:52 (1476500272) [19383.338068] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 19:57:54 (1476500274) [19385.397836] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 19:57:56 (1476500276) [19385.910029] Lustre: *** cfs_fail_loc=20e, val=0*** [19388.847777] Lustre: *** cfs_fail_loc=20e, val=0*** [19388.856664] Lustre: Skipped 1 previous similar message [19395.870429] Lustre: *** cfs_fail_loc=20e, val=0*** [19395.879229] Lustre: Skipped 1 previous similar message [19397.960912] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 19:58:08 (1476500288) [19403.904194] Lustre: *** cfs_fail_loc=20e, val=0*** [19403.912053] Lustre: Skipped 3 previous similar messages [19409.926172] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 19:58:20 (1476500300) [19412.611317] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 19:58:23 (1476500303) [19423.500741] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 19:58:34 (1476500314) [19425.280134] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 19:58:36 (1476500316) [19427.093883] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 19:58:37 (1476500317) [19428.939638] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 19:58:39 (1476500319) [19429.347282] Lustre: DEBUG MARKER: SKIP: sanity test_119b skipping 2-stripe test [19430.981155] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 19:58:41 (1476500321) [19432.873089] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 19:58:43 (1476500323) [19434.639681] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [19438.252721] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 19:58:49 (1476500329) [19440.615576] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 19:58:51 (1476500331) [19442.969517] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 19:58:53 (1476500333) [19445.313814] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 19:58:56 (1476500336) [19447.667440] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 19:58:58 (1476500338) [19455.195517] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 19:59:06 (1476500346) [19462.583468] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 19:59:13 (1476500353) [19493.997712] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 19:59:44 (1476500384) [19495.730378] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 19:59:46 (1476500386) [19496.121736] Lustre: DEBUG MARKER: testing on UP system. Performance may be not as good as expected. [19496.876123] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [19497.357252] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [19500.384106] Lustre: DEBUG MARKER: ls 1000 files without statahead: 2 sec [19501.109073] Lustre: DEBUG MARKER: ls 1000 files with statahead: 0 sec [19526.816208] Lustre: DEBUG MARKER: ls 10000 files without statahead: 15 sec [19529.497904] Lustre: DEBUG MARKER: ls 10000 files with statahead: 2 sec [19529.892970] Lustre: DEBUG MARKER: ls done [19544.583991] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 14 seconds [19544.992165] Lustre: DEBUG MARKER: rm done [19546.618525] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 20:00:37 (1476500437) [19550.049638] Lustre: DEBUG MARKER: ls done [19552.404413] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 20:00:43 (1476500443) [19552.826843] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [19555.972105] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880036dd0000 [19556.369714] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880036dd0000 [19556.755564] Lustre: DEBUG MARKER: LRU=2003 [19557.183447] Lustre: DEBUG MARKER: LIMIT=1596099 [19557.581124] Lustre: DEBUG MARKER: LVF=956225 [19558.000129] Lustre: DEBUG MARKER: OLD_LVF=1 [19558.404975] Lustre: DEBUG MARKER: Sleep 50 sec [19608.860473] Lustre: DEBUG MARKER: Dropped 752 locks in 50s [19609.308636] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [19612.646695] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 20:01:43 (1476500503) [19615.773698] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [19620.538990] Lustre: DEBUG MARKER: ls -la time: 5 seconds [19620.927200] Lustre: DEBUG MARKER: lru_size = 100 [19625.618021] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [19630.148735] Lustre: DEBUG MARKER: ls -la time: 4 seconds [19630.551271] Lustre: DEBUG MARKER: lru_size = 2006 [19630.924102] Lustre: DEBUG MARKER: ls -la is 20% faster with lru resize enabled [19633.701334] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 20:02:04 (1476500524) [19655.773983] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 20:02:26 (1476500546) [19657.571040] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 20:02:28 (1476500548) [19659.181809] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 20:02:29 (1476500549) [19661.208715] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 20:02:31 (1476500551) [19663.028417] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 20:02:33 (1476500553) [19664.686080] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 20:02:35 (1476500555) [19768.113878] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 20:04:18 (1476500658) [19769.819223] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (1-stripe file) ========================================================== 20:04:20 (1476500660) [19770.217605] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping FIEMAP on 1-stripe file [19771.826328] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 20:04:22 (1476500662) [19772.232995] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping FIEMAP on 2-stripe file [19773.834455] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 20:04:24 (1476500664) [19774.250125] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [19775.860308] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 20:04:26 (1476500666) [19776.259488] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping continuation FIEMAP test [19777.939189] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 20:04:28 (1476500668) [19779.714753] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 20:04:30 (1476500670) [19781.468024] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 20:04:32 (1476500672) [19783.165079] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 20:04:33 (1476500673) [19784.796166] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 20:04:35 (1476500675) [19786.455288] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 20:04:37 (1476500677) [19792.384806] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 20:04:43 (1476500683) [19796.524417] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 20:04:47 (1476500687) [19815.543914] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 20:05:06 (1476500706) [19821.535889] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 20:05:12 (1476500712) [19824.704014] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc ================================ 20:05:15 (1476500715) [19830.453489] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc =========================== 20:05:21 (1476500721) [19833.423710] LustreError: 22568:0:(ldlm_resource.c:109:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [19833.438975] LustreError: 22568:0:(ldlm_resource.c:109:seq_watermark_write()) Skipped 9 previous similar messages [19833.494855] LustreError: 22600:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Can't parse user input, rc = -14 [19833.509048] LustreError: 22600:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Skipped 14 previous similar messages [19833.537846] LustreError: 22610:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) invalid value written [19833.537850] LustreError: 22610:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) Skipped 6 previous similar messages [19833.676682] LustreError: 22690:0:(genops.c:1519:obd_export_evict_by_uuid()) lustre-OST0000: can't disconnect : no exports found [19833.693639] LustreError: 22690:0:(genops.c:1519:obd_export_evict_by_uuid()) Skipped 3 previous similar messages [19833.852799] LustreError: 22819:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) srv-lustre-OST0000: invalid FID sequence width: rc = -14 [19833.871768] LustreError: 22819:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) Skipped 4 previous similar messages [19833.892290] LustreError: 22825:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) cannot add nodemap config to non-existing MGS. [19833.909620] LustreError: 22825:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) Skipped 1 previous similar message [19833.932902] LustreError: 22833:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [19833.955410] Lustre: 22850:0:(libcfs_string.c:131:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [19833.955413] Lustre: 22850:0:(libcfs_string.c:131:cfs_str2mask()) Skipped 2 previous similar messages [19833.957733] Lustre: 0 MB is too small for debug buffer size, setting it to 128 MB. [19835.325463] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [19835.340865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [19847.330634] Lustre: 23238:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476500731/real 1476500731] req@ffff8803f5a2d700 x1548202901778448/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476500738 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19847.372067] Lustre: 23238:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [19847.386517] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [19854.285169] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [19854.298071] Lustre: Skipped 13 previous similar messages [19857.413430] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620011fb0a to 0x2029536200389ee3 [19857.433622] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19862.863526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37146 to 0x0:37313 [19863.416951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [19867.418132] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [19867.433436] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [19879.423180] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [19879.855479] Lustre: server umount lustre-OST0000 complete [19902.998469] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [19903.022473] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [19904.401425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37146 to 0x0:37345 [19908.384251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [19915.323839] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 20:06:45 (1476500805) [19931.672072] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 20:07:02 (1476500822) [19961.791241] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 20:07:32 (1476500852) [19965.523345] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 20:07:36 (1476500856) [19967.924569] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 20:07:38 (1476500858) [19974.748582] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 20:07:45 (1476500865) [19976.452915] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 20:07:47 (1476500867) [19978.328744] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 20:07:48 (1476500868) [19980.349862] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 20:07:50 (1476500870) [19983.586062] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 20:07:54 (1476500874) [19984.320135] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [19985.931700] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 20:07:56 (1476500876) [19988.034344] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 20:07:58 (1476500878) [19990.384503] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 20:08:01 (1476500881) [19992.321399] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 20:08:02 (1476500882) [19994.138091] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 20:08:04 (1476500884) [20050.716709] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 20:09:01 (1476500941) [20054.755473] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 20:09:05 (1476500945) [20058.523776] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 20:09:09 (1476500949) [20062.298523] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 20:09:12 (1476500952) [20066.248243] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 20:09:16 (1476500956) [20077.378915] Lustre: DEBUG MARKER: SKIP: sanity test_155e max available OST size needs > 1310720 KB [20079.693932] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 20:09:30 (1476500970) [20083.168291] Lustre: DEBUG MARKER: SKIP: sanity test_155f max available OST size needs > 1310720 KB [20085.424274] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 20:09:35 (1476500975) [20088.868862] Lustre: DEBUG MARKER: SKIP: sanity test_155g max available OST size needs > 1310720 KB [20091.144081] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 20:09:41 (1476500981) [20094.672661] Lustre: DEBUG MARKER: SKIP: sanity test_155h max available OST size needs > 1310720 KB [20096.958580] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 20:09:47 (1476500987) [20099.156129] Lustre: DEBUG MARKER: Turn on read and write cache [20100.255091] Lustre: DEBUG MARKER: Write data and read it back. [20100.646884] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [20101.772265] Lustre: DEBUG MARKER: cache hits:: before: 36, after: 39 [20102.179453] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20102.943035] Lustre: DEBUG MARKER: cache hits:: before: 39, after: 42 [20103.330742] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [20104.449393] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20105.574427] Lustre: DEBUG MARKER: cache hits:: before: 42, after: 45 [20105.978896] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [20106.811354] Lustre: DEBUG MARKER: cache hits:: before: 45, after: 45 [20107.208482] Lustre: DEBUG MARKER: Write data and read it back. [20107.615372] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [20108.808522] Lustre: DEBUG MARKER: cache hits:: before: 45, after: 48 [20109.220780] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [20110.000187] Lustre: DEBUG MARKER: cache hits:: before: 48, after: 48 [20110.405543] Lustre: DEBUG MARKER: Turn off read and write cache [20111.558096] Lustre: DEBUG MARKER: Write data and read it back [20111.955575] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [20113.218329] Lustre: DEBUG MARKER: cache hits:: before: 48, after: 48 [20113.618040] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [20114.769571] Lustre: DEBUG MARKER: Write data and read it back [20115.179061] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [20116.374478] Lustre: DEBUG MARKER: cache hits:: before: 48, after: 48 [20116.774369] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [20117.929408] Lustre: DEBUG MARKER: cache hits:: before: 48, after: 51 [20120.219374] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 20:10:10 (1476501010) [20128.510782] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [20128.525285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [20135.513414] Lustre: 23252:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476501019/real 1476501019] req@ffff8803f57b6300 x1548202901781408/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476501026 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20135.551955] Lustre: 23252:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [20135.565456] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [20135.585949] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620038a079 to 0x2029536200421cd1 [20136.398040] Lustre: lustre-OST0000: deleting orphan objects from 0x0:38983 to 0x0:39009 [20140.587716] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20140.928324] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 20:10:31 (1476501031) [20144.094295] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 20:10:34 (1476501034) [20148.078789] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 20:10:38 (1476501038) [20148.509980] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [20150.103172] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 20:10:40 (1476501040) [20155.057361] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 20:10:45 (1476501045) [20155.450364] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [20157.032327] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 20:10:47 (1476501047) [20160.197964] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 20:10:50 (1476501050) [20161.993090] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 20:10:52 (1476501052) [20162.386881] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [20164.005408] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 20:10:54 (1476501054) [20168.642400] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 20:10:59 (1476501059) [20169.050522] Lustre: DEBUG MARKER: creating a 10 Mb file [20170.891005] Lustre: DEBUG MARKER: starting reads [20171.396784] Lustre: DEBUG MARKER: truncating the file [20171.818779] Lustre: DEBUG MARKER: killing dd [20172.216963] Lustre: DEBUG MARKER: removing the temporary file [20173.890539] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 20:11:04 (1476501064) [20177.601216] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 20:11:08 (1476501068) [20182.347668] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 20:11:12 (1476501072) [20185.710684] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 20:11:16 (1476501076) [20186.084799] Lustre: Echo OBD driver; http://www.lustre.org/ [20191.927722] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ============================= 20:11:22 (1476501082) [20192.841590] Lustre: Echo OBD driver; http://www.lustre.org/ [20201.575130] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 20:11:32 (1476501092) [20218.490932] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 20:11:48 (1476501108) [20235.173999] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========================= 20:12:05 (1476501125) [20237.869709] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =============================================================== 20:12:08 (1476501128) [20239.857324] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ==================================== 20:12:10 (1476501130) [20241.800679] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ================================================ 20:12:12 (1476501132) [20253.092964] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap =================================================== 20:12:23 (1476501143) [20254.876687] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 20:12:25 (1476501145) [20257.019831] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ============================================================ 20:12:27 (1476501147) [20260.123023] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ======================================================== 20:12:30 (1476501150) [20264.067817] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ======================================= 20:12:34 (1476501154) [20266.602040] Lustre: DEBUG MARKER: == sanity test 200: OST pools ======================================================================== 20:12:37 (1476501157) [20308.697581] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ================================================= 20:13:19 (1476501199) [20310.402817] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset =========================================== 20:13:20 (1476501200) [20312.134124] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset =========================================== 20:13:22 (1476501202) [20313.802435] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ============================================= 20:13:24 (1476501204) [20315.506889] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ===================================================== 20:13:25 (1476501205) [20317.293122] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset =============================================== 20:13:27 (1476501207) [20319.049724] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset =============================================== 20:13:29 (1476501209) [20320.746986] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ================================================= 20:13:31 (1476501211) [20322.504717] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats ================================================================= 20:13:32 (1476501212) [20334.449188] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [20334.858432] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.31601 [20335.601777] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [20335.986201] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.15616 [20336.738343] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [20337.133044] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.112 [20337.847317] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [20338.282151] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.30444 [20339.017246] Lustre: DEBUG MARKER: Test: /bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [20339.436079] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.9688 [20340.196139] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [20340.591129] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.12587 [20341.664737] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [20342.049370] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.6818 [20343.328637] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [20343.750848] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.445 [20344.523289] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [20344.903106] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.19726 [20345.991841] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [20346.386074] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.2869 [20347.130774] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [20347.539532] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.32750 [20366.747796] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug =============================================== 20:14:17 (1476501257) [20368.484482] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse =================================================== 20:14:18 (1476501258) [20370.149677] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ====================================================== 20:14:20 (1476501260) [20371.891545] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open =================================================================== 20:14:22 (1476501262) [20380.704154] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [20380.719244] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [20387.706681] Lustre: 23240:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476501271/real 1476501271] req@ffff8803ec106c00 x1548202901785344/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476501278 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20387.747000] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [20397.773569] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200421cd1 to 0x20295362005603a4 [20399.810340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:53056 to 0x0:53409 [20400.740041] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [20402.775963] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20413.410316] LustreError: 23356:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [20424.416506] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20429.920421] Lustre: lustre-OST0000: deleting orphan objects from 0x0:53056 to 0x0:53441 [20430.539043] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [20433.414920] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly =========================== 20:15:23 (1476501323) [20433.431812] LustreError: 23356:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [20433.432984] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362005603a4 to 0x2029536200560659 [20442.547749] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 20:15:32 (1476501332) [20445.798855] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 20:15:36 (1476501336) [20447.672576] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 20:15:38 (1476501338) [20452.125942] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 20:15:42 (1476501342) [20453.828859] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 20:15:44 (1476501344) [20458.316158] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 20:15:48 (1476501348) [20460.953590] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 20:15:51 (1476501351) [20461.352723] Lustre: DEBUG MARKER: creating a 10 Mb file [20462.621637] Lustre: DEBUG MARKER: starting reads [20463.246543] Lustre: DEBUG MARKER: truncating the file [20463.641759] Lustre: DEBUG MARKER: killing dd [20464.026145] Lustre: DEBUG MARKER: removing the temporary file [20465.665415] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ================== 20:15:56 (1476501356) [20467.809062] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST =========================== 20:15:58 (1476501358) [20475.039445] Lustre: *** cfs_fail_loc=229, val=0*** [20475.048268] Lustre: Skipped 1 previous similar message [20480.065835] Lustre: *** cfs_fail_loc=229, val=0*** [20485.092276] Lustre: *** cfs_fail_loc=229, val=0*** [20504.591632] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 20:16:34 (1476501394) [20534.391022] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 20:17:04 (1476501424) [20536.106654] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 20:17:06 (1476501426) [20537.841966] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 20:17:08 (1476501428) [20539.690193] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ================================================== 20:17:10 (1476501430) [20539.760342] Lustre: lustre-OST0000: Client 331ca3dd-38f3-9094-b8b4-62eb25833a59 (at 10.2.4.36@tcp) reconnecting [20539.760385] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [20539.760387] Lustre: Skipped 16 previous similar messages [20541.677777] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ================================================== 20:17:12 (1476501432) [20541.757491] Lustre: lustre-OST0000: Client 331ca3dd-38f3-9094-b8b4-62eb25833a59 (at 10.2.4.36@tcp) reconnecting [20542.743798] LustreError: 34234:0:(ldlm_lib.c:3238:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804278e0c50 x1548223287987808/t0(0) o3->331ca3dd-38f3-9094-b8b4-62eb25833a59@10.2.4.36@tcp:-1/-1 lens 608/432 e 0 to 0 dl 1476501475 ref 1 fl Interpret:/0/0 rc 0/0 [20542.780265] Lustre: lustre-OST0000: Bulk IO read error with 331ca3dd-38f3-9094-b8b4-62eb25833a59 (at 10.2.4.36@tcp), client will retry: rc -110 [20544.485750] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 20:17:14 (1476501434) [20560.859137] Lustre: *** cfs_fail_loc=520, val=2147483648*** [20577.818992] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 20:17:48 (1476501468) [20591.414828] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 20:18:01 (1476501481) [20605.295377] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ================================= 20:18:15 (1476501495) [20606.988694] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 20:18:17 (1476501497) [20607.406719] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [20609.394145] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ================================== 20:18:19 (1476501499) [20611.043624] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ===================================================== 20:18:21 (1476501501) [20662.334367] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ================================== 20:19:12 (1476501552) [20706.564826] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [20706.580276] LustreError: Skipped 1 previous similar message [20706.589749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [20706.614401] Lustre: Skipped 1 previous similar message [20716.569396] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [20716.569534] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20716.614373] LustreError: Skipped 1 previous similar message [20716.624877] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200560659 to 0x2029536200717a05 [20718.691733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76862 to 0x0:78433 [20730.950417] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 20:20:21 (1476501621) [20818.282612] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 20:21:48 (1476501708) [20819.006802] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [20820.636016] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ==================== 20:21:50 (1476501710) [20821.044717] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [20822.703760] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =============================================================== 20:21:52 (1476501712) [20823.119674] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [20824.755003] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed ============================= 20:21:54 (1476501714) [20825.168072] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [20826.785720] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory ===================================================== 20:21:56 (1476501716) [20827.181176] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [20828.816470] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files =============================================== 20:21:59 (1476501719) [20829.218492] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [20830.861304] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files ============================================== 20:22:01 (1476501721) [20831.268634] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [20832.928655] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ==================================================== 20:22:03 (1476501723) [20833.335884] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [20835.007208] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ============================================================= 20:22:05 (1476501725) [20835.419721] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [20837.085024] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 20:22:07 (1476501727) [20837.494475] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [20839.111951] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ================ 20:22:09 (1476501729) [20842.152215] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 20:22:12 (1476501732) [20843.527740] Lustre: 34273:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [20843.547675] Lustre: 34273:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [20843.561284] Lustre: 34273:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [20843.575114] Lustre: 34273:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [20843.589495] Lustre: 34273:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [20843.602321] Lustre: 34273:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [20844.569181] Lustre: 34273:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [20844.588346] Lustre: 34273:0:(osd_internal.h:1079:osd_trans_exec_op()) Skipped 1 previous similar message [20844.601682] Lustre: 34273:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [20844.614722] Lustre: 34273:0:(osd_handler.c:1639:osd_trans_dump_creds()) Skipped 1 previous similar message [20844.627874] Lustre: 34273:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [20844.641036] Lustre: 34273:0:(osd_handler.c:1646:osd_trans_dump_creds()) Skipped 1 previous similar message [20844.653856] Lustre: 34273:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [20844.667501] Lustre: 34273:0:(osd_handler.c:1656:osd_trans_dump_creds()) Skipped 1 previous similar message [20844.680068] Lustre: 34273:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [20844.692149] Lustre: 34273:0:(osd_handler.c:1663:osd_trans_dump_creds()) Skipped 1 previous similar message [20844.704504] Lustre: 34273:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [20844.716596] Lustre: 34273:0:(osd_handler.c:1670:osd_trans_dump_creds()) Skipped 1 previous similar message [20845.952321] Lustre: 34273:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [20845.971220] Lustre: 34273:0:(osd_internal.h:1079:osd_trans_exec_op()) Skipped 1 previous similar message [20845.984435] Lustre: 34273:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [20845.997333] Lustre: 34273:0:(osd_handler.c:1639:osd_trans_dump_creds()) Skipped 1 previous similar message [20846.010356] Lustre: 34273:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [20846.023401] Lustre: 34273:0:(osd_handler.c:1646:osd_trans_dump_creds()) Skipped 1 previous similar message [20846.036099] Lustre: 34273:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [20846.049655] Lustre: 34273:0:(osd_handler.c:1656:osd_trans_dump_creds()) Skipped 1 previous similar message [20846.062119] Lustre: 34273:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [20846.074082] Lustre: 34273:0:(osd_handler.c:1663:osd_trans_dump_creds()) Skipped 1 previous similar message [20846.086344] Lustre: 34273:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [20846.098331] Lustre: 34273:0:(osd_handler.c:1670:osd_trans_dump_creds()) Skipped 1 previous similar message [20849.219512] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount ============================================== 20:22:19 (1476501739) [20851.452685] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 20:22:21 (1476501741) [20853.476926] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 20:22:23 (1476501743) [20855.464207] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 20:22:25 (1476501745) [20856.032004] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.46 too old [20857.704964] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 20:22:27 (1476501747) [20861.717580] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ================================================ 20:22:31 (1476501751) [20863.484069] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls ============================== 20:22:33 (1476501753) [20865.176183] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ======================================================== 20:22:35 (1476501755) [20867.334752] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test ==================================================================== 20:22:37 (1476501757) [20890.737297] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ======================================= 20:23:00 (1476501780) [20899.000697] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 20:23:09 (1476501789) [20904.530432] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ===================== 20:23:14 (1476501794) [20904.939535] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [20906.572271] Lustre: DEBUG MARKER: == sanity test 241: bio vs dio ======================================================================= 20:23:16 (1476501796) [20919.182602] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ====================================================================== 20:23:29 (1476501809) [20924.430013] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 20:23:34 (1476501814) [20926.738850] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ========================================================= 20:23:36 (1476501816) [20948.919408] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests =================================================== 20:23:59 (1476501839) [20964.305375] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ============================= 20:24:14 (1476501854) [20965.960063] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length ================================ 20:24:16 (1476501856) [20967.032693] Lustre: *** cfs_fail_loc=234, val=0*** [20967.041314] Lustre: Skipped 25 previous similar messages [20968.637628] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ========================================================= 20:24:18 (1476501858) [20970.756413] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ================================================ 20:24:20 (1476501860) [20972.455578] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside root =================================================== 20:24:22 (1476501862) [20974.338862] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside root ==================================================== 20:24:24 (1476501864) [20976.186371] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ============================================================= 20:24:26 (1476501866) [20977.938617] Lustre: DEBUG MARKER: == sanity test 248: fast read verification =========================================================== 20:24:28 (1476501868) [20987.709980] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ========================================================= 20:24:37 (1476501877) [20989.692847] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ============================================================ 20:24:39 (1476501879) [20991.375762] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================== 20:24:41 (1476501881) [20993.070920] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ============================================================= 20:24:43 (1476501883) [20996.066370] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ==================================================== 20:24:46 (1476501886) [21021.192439] Lustre: lustre-OST0000: haven't heard from client ee07d3e1-99a7-e864-f24e-e20843d3498c (at 10.2.4.36@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp ffff880823b7ac00, cur 1476501911 expire 1476501881 last 1476501861 [21021.224334] Lustre: Skipped 1 previous similar message [21087.158720] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ============================================================= 20:26:17 (1476501977) [21090.652631] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 20:26:20 (1476501980) [21094.503153] LustreError: 23367:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 sleeping for 4000ms [21098.520156] LustreError: 23367:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 awake [21098.554585] LustreError: 23367:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 sleeping for 4000ms [21099.572520] sh (53753): drop_caches: 3 [21102.571114] LustreError: 23367:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 237 awake [21145.211048] sh (53794): drop_caches: 3 [21166.629955] sh (53834): drop_caches: 3 [21290.224432] sh (53879): drop_caches: 3 [21307.934376] sh (53919): drop_caches: 3 [21431.195873] sh (53963): drop_caches: 3 [21449.800461] sh (54002): drop_caches: 3 [21573.646430] sh (54048): drop_caches: 3 [21591.896361] sh (54087): drop_caches: 3 [21715.740643] sh (54132): drop_caches: 3 [21739.731511] sh (54172): drop_caches: 3 [21863.781385] sh (54216): drop_caches: 3 [21888.013402] sh (54257): drop_caches: 3 [22011.561670] sh (54300): drop_caches: 3 [22038.001295] sh (54341): drop_caches: 3 [22161.874291] sh (54385): drop_caches: 3 [22187.898718] sh (54427): drop_caches: 3 [22310.637487] sh (54471): drop_caches: 3 [22335.540442] sh (54512): drop_caches: 3 [22458.702054] sh (54555): drop_caches: 3 [22484.779158] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ================================================= 20:49:34 (1476503374) [22489.624901] sh (54831): drop_caches: 3 [22495.093306] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state =================================== 20:49:44 (1476503384) [22497.433118] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [22497.447432] LustreError: Skipped 1 previous similar message [22497.455845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [22504.435703] Lustre: 23243:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476503387/real 1476503387] req@ffff880416b82100 x1548202901807712/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476503394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22504.473607] Lustre: 23243:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [22504.486904] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [22506.158301] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [22506.170127] Lustre: Skipped 14 previous similar messages [22514.511501] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200717a05 to 0x202953620097d1e1 [22519.513805] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [22520.619118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:107485 to 0x0:107713 [22523.442782] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ========================================================= 20:50:12 (1476503412) [22529.518679] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [22529.532928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [22534.520986] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [22534.541487] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620097d1e1 to 0x202953620097d609 [22536.413095] Lustre: lustre-OST0000: deleting orphan objects from 0x0:107723 to 0x0:107745 [22538.949895] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ============================================================= 20:50:28 (1476503428) [22539.523585] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [22540.656665] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test =================================================== 20:50:30 (1476503430) [22541.395522] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [22543.872798] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir =============================================== 20:50:33 (1476503433) [22544.617248] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [22546.231853] Lustre: DEBUG MARKER: == sanity test 300c: chown [22547.043106] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [22548.674608] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ==================================== 20:50:38 (1476503438) [22549.413345] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [22551.043922] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ============================================ 20:50:40 (1476503440) [22551.769880] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [22553.366198] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ============================================ 20:50:42 (1476503442) [22554.093124] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [22555.790830] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ============================ 20:50:45 (1476503445) [22556.539097] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [22558.182144] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory =========================== 20:50:47 (1476503447) [22558.913806] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [22560.535508] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory =============================== 20:50:49 (1476503449) [22561.259617] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [22562.866391] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ======================================================== 20:50:52 (1476503452) [22563.597965] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [22565.236706] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ==================================================== 20:50:54 (1476503454) [22565.976532] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [22567.589597] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 20:50:56 (1476503456) [22568.322977] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [22569.966027] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS =========================================== 20:50:59 (1476503459) [22572.077416] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 20:51:01 (1476503461) [22572.825293] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [22574.488424] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [22574.917943] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================== 20:51:04 (1476503464) [22575.329566] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [22576.915962] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ================================== 20:51:06 (1476503466) [22577.336756] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [22578.961226] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ========================================================= 20:51:08 (1476503468) [22579.356901] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [22580.966019] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open =============================== 20:51:10 (1476503470) [22581.351802] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [22583.017061] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ===================================== 20:51:12 (1476503472) [22583.428301] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [22585.070398] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 20:51:14 (1476503474) [22588.987045] Lustre: DEBUG MARKER: sanity test_311: @@@@@@ FAIL: setstripe failed [22592.972129] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 20:51:22 (1476503482) [22593.365914] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [22594.991120] Lustre: DEBUG MARKER: == sanity test 399: fake write should not be slower than normal write ================================ 20:51:24 (1476503484) [22595.510214] Lustre: DEBUG MARKER: sanity test_399: @@@@@@ FAIL: real dd writing error [22596.815161] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link ================================== 20:51:26 (1476503486) [22599.996102] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ================================================ 20:51:29 (1476503489) [22601.713221] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ================== 20:51:31 (1476503491) [22603.382039] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ============================== 20:51:32 (1476503492) [22604.996856] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. =================== 20:51:34 (1476503494) [22606.663193] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' =========================== 20:51:36 (1476503496) [22608.281353] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 20:51:37 (1476503497) [22610.241558] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 20:51:39 (1476503499) [22610.838611] Lustre: DEBUG MARKER: sanity test_403: @@@@@@ FAIL: wrong nlink count: [22612.192247] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 20:51:41 (1476503501) [22613.581155] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 10.2.4.47@tcp) reconnecting [22613.594999] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [22613.607707] Lustre: Skipped 5 previous similar messages [22613.639968] Lustre: lustre-OST0000: deleting orphan objects from 0x0:108747 to 0x0:109025 [22615.572999] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests =================================================== 20:51:44 (1476503504) [22616.383034] Lustre: DEBUG MARKER: sanity test_405: @@@@@@ FAIL: One layout swap locked test failed [22617.822753] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ================================================== 20:51:47 (1476503507) [22618.225175] Lustre: DEBUG MARKER: SKIP: sanity test_406 needs >= 2 MDTs [22619.837417] Lustre: DEBUG MARKER: == sanity test 407: transaction fail should cause operation fail ===================================== 20:51:49 (1476503509) [22620.254936] Lustre: DEBUG MARKER: SKIP: sanity test_407 needs >= 2 MDTs [22621.874764] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ==================================== 20:51:51 (1476503511) [22624.021414] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 20:51:53 (1476503513) [22629.567092] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [22629.582465] LustreError: Skipped 1 previous similar message [22629.591900] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [22641.572164] Lustre: 23240:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476503524/real 1476503524] req@ffff8803f57b6f00 x1548202901809056/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476503531 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22641.613570] Lustre: 23240:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [22645.364890] Lustre: server umount lustre-OST0000 complete [22669.429960] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [22669.464169] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [22679.418333] Lustre: lustre-OST0000: deleting orphan objects from 0x0:108747 to 0x0:109025 [22685.397836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [22692.409192] Lustre: DEBUG MARKER: == sanity test complete, duration 5053 sec =========================================================== 20:53:01 (1476503581) [29433.498829] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [29433.513941] Lustre: Skipped 3 previous similar messages [29533.838136] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 22:46:59 PDT 2016 [29537.658829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [29540.328478] Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b [29540.709023] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o [29541.162171] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 22:47:07 (1476510427) [29542.289740] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 22:47:08 (1476510428) [29543.446634] Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 22:47:09 (1476510429) [29545.207071] Lustre: DEBUG MARKER: == sanity test complete, duration 11 sec == 22:47:11 (1476510431) [29588.333200] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Oct 14 22:47:54 PDT 2016 [29592.123509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [29594.593020] Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b [29594.966650] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o [29595.422547] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 22:48:01 (1476510481) [29596.817983] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 22:48:02 (1476510482) [29597.975392] Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 22:48:03 (1476510483) [29599.110822] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 22:48:05 (1476510485) [29600.299832] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ====================== 22:48:06 (1476510486) [29601.490761] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ======================= 22:48:07 (1476510487) [29602.678126] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) == 22:48:08 (1476510488) [29603.878587] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============== 22:48:09 (1476510489) [29605.207990] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ==== 22:48:11 (1476510491) [29606.364583] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ==== 22:48:12 (1476510492) [29607.555009] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ==== 22:48:13 (1476510493) [29608.783551] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? == 22:48:14 (1476510494) [29610.118641] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) == 22:48:16 (1476510496) [29611.360283] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ====== 22:48:17 (1476510497) [29612.529788] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ======= 22:48:18 (1476510498) [29613.733633] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ========= 22:48:19 (1476510499) [29614.918328] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================== 22:48:20 (1476510500) [29616.086404] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ======== 22:48:22 (1476510502) [29617.250207] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============== 22:48:23 (1476510503) [29618.706946] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ========== 22:48:24 (1476510504) [29619.889766] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ========== 22:48:25 (1476510505) [29621.026097] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ========= 22:48:26 (1476510506) [29622.171717] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ============ 22:48:28 (1476510508) [29623.356394] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f =================== 22:48:29 (1476510509) [29624.540701] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ==================== 22:48:30 (1476510510) [29625.716613] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ================ 22:48:31 (1476510511) [29626.920980] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ====== 22:48:32 (1476510512) [29628.065426] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ========================== 22:48:34 (1476510514) [29629.251570] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ====== 22:48:35 (1476510515) [29630.411553] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================== 22:48:36 (1476510516) [29631.616758] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries == 22:48:37 (1476510517) [29633.944357] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug == 22:48:39 (1476510519) [29635.404070] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink == 22:48:41 (1476510521) [29636.870728] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled =========================== 22:48:42 (1476510522) [29638.099995] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========== 22:48:44 (1476510524) [29639.290440] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink == 22:48:45 (1476510525) [29653.249222] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [29653.265023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [29659.306674] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [29659.319901] Lustre: Skipped 1 previous similar message [29660.293765] Lustre: 23254:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476510539/real 1476510539] req@ffff880802fa8600 x1548202901986832/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476510546 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29660.335989] Lustre: 23254:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [29660.350611] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [29660.373585] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200988890 to 0x2029536200a0a83f [29660.394415] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29671.805582] Lustre: lustre-OST0000: deleting orphan objects from 0x0:109051 to 0x0:109089 [29672.901326] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir == 22:49:18 (1476510558) [29673.946983] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [29675.106046] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 22:49:21 (1476510561) [29680.382295] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [29680.397982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [29687.384917] Lustre: 23261:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476510566/real 1476510566] req@ffff880802fa9800 x1548202901987264/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476510573 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29687.426812] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [29697.475796] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200a0a83f to 0x2029536200a0aae6 [29697.496623] Lustre: MGC10.2.4.47@tcp: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [29697.509881] Lustre: Skipped 3 previous similar messages [29702.456990] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29702.796740] Lustre: lustre-OST0000: deleting orphan objects from 0x0:109091 to 0x0:109121 [29703.415133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [29706.458677] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ================================ 22:49:52 (1476510592) [29707.703750] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ============= 22:49:53 (1476510593) [29709.016723] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ================ 22:49:54 (1476510594) [29710.152517] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ==== 22:49:56 (1476510596) [29711.332654] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================ 22:49:57 (1476510597) [29712.487588] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... ============================= 22:49:58 (1476510598) [29712.883813] Lustre: DEBUG MARKER: 1 done [29713.279907] Lustre: DEBUG MARKER: 2 done [29713.677524] Lustre: DEBUG MARKER: 3 done [29714.843554] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================== 22:50:00 (1476510600) [29716.071134] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user ================= 22:50:01 (1476510601) [29716.465661] perf interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 [29717.561843] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============================ 22:50:03 (1476510603) [29718.767174] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ============================ 22:50:04 (1476510604) [29719.962075] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target == 22:50:05 (1476510605) [29721.172723] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target == 22:50:07 (1476510607) [29722.341766] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target == 22:50:08 (1476510608) [29723.522652] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 22:50:09 (1476510609) [29724.717368] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ======== 22:50:10 (1476510610) [29725.931956] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ====== 22:50:11 (1476510611) [29727.133372] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ======== 22:50:13 (1476510613) [29728.321608] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e == 22:50:14 (1476510614) [29729.497067] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a == 22:50:15 (1476510615) [29730.678199] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================== 22:50:16 (1476510616) [29731.881232] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ========= 22:50:17 (1476510617) [29733.034583] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ========================== 22:50:18 (1476510618) [29734.208877] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =========== 22:50:20 (1476510620) [29735.348649] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) == 22:50:21 (1476510621) [29736.533348] Lustre: DEBUG MARKER: SKIP: sanity test_24o skipping SLOW test 24o [29736.927466] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b == 22:50:22 (1476510622) [29738.124977] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ===== 22:50:24 (1476510624) [29739.332095] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ======= 22:50:25 (1476510625) [29740.496585] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c === 22:50:26 (1476510626) [29741.665374] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a === 22:50:27 (1476510627) [29742.862997] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 22:50:28 (1476510628) [29744.109077] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) == 22:50:30 (1476510630) [29744.514525] Lustre: DEBUG MARKER: SKIP: sanity test_24v not enough free inodes 98597 required 100000 [29745.686621] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb == 22:50:31 (1476510631) [29746.913580] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link == 22:50:32 (1476510632) [29747.289872] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [29748.447947] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed == 22:50:34 (1476510634) [29748.824276] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [29750.087178] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. == 22:50:35 (1476510635) [29767.126109] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries == 22:50:53 (1476510653) [29767.500216] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [29768.685725] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir == 22:50:54 (1476510654) [29769.080773] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [29770.255593] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [29770.672787] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link == 22:50:56 (1476510656) [29771.066114] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [29772.213508] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ================= 22:50:58 (1476510658) [29773.393719] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ================= 22:50:59 (1476510659) [29774.578697] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ========================= 22:51:00 (1476510660) [29776.057217] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ======== 22:51:01 (1476510661) [29777.236446] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================== 22:51:03 (1476510663) [29778.433766] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ======== 22:51:04 (1476510664) [29779.567722] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ======== 22:51:05 (1476510665) [29780.731636] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink === 22:51:06 (1476510666) [29781.960348] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ==================================== 22:51:07 (1476510667) [29782.403364] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [29783.578059] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 22:51:09 (1476510669) [29783.967112] Lustre: DEBUG MARKER: SKIP: sanity test_27b skipping 2-stripe test [29785.131955] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================== 22:51:11 (1476510671) [29786.301399] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ======== 22:51:12 (1476510672) [29787.855806] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) == 22:51:13 (1476510673) [29789.102054] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects == 22:51:14 (1476510674) [29790.270313] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects == 22:51:16 (1476510676) [29791.475960] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) == 22:51:17 (1476510677) [29792.682394] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps =============== 22:51:18 (1476510678) [29793.905925] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) == 22:51:19 (1476510679) [29795.091720] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [29795.498538] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ==================== 22:51:21 (1476510681) [29795.894945] Lustre: DEBUG MARKER: SKIP: sanity test_27n too few OSTs [29797.040972] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ====== 22:51:22 (1476510682) [29797.430025] Lustre: DEBUG MARKER: SKIP: sanity test_27o too few OSTs [29798.592321] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ===== 22:51:24 (1476510684) [29798.977306] Lustre: DEBUG MARKER: SKIP: sanity test_27p too few OSTs [29800.124404] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ===== 22:51:25 (1476510685) [29800.512622] Lustre: DEBUG MARKER: SKIP: sanity test_27q too few OSTs [29801.662088] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) === 22:51:27 (1476510687) [29802.049610] Lustre: DEBUG MARKER: SKIP: sanity test_27r too few OSTs [29803.229303] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) == 22:51:29 (1476510689) [29804.401884] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly == 22:51:30 (1476510690) [29805.571351] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ============ 22:51:31 (1476510691) [29805.954965] Lustre: DEBUG MARKER: SKIP: sanity test_27u too few OSTs [29807.150996] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST =================== 22:51:32 (1476510692) [29807.533425] Lustre: DEBUG MARKER: SKIP: sanity test_27v too few OSTs [29808.691198] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option == 22:51:34 (1476510694) [29809.866551] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options == 22:51:35 (1476510695) [29810.254808] Lustre: DEBUG MARKER: SKIP: sanity test_27wa skipping multiple stripe count/offset test [29811.850822] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 22:51:37 (1476510697) [29812.252680] Lustre: DEBUG MARKER: SKIP: sanity test_27x 1 < 2 OSTs [29813.402602] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive == 22:51:39 (1476510699) [29813.801626] Lustre: DEBUG MARKER: SKIP: sanity test_27y 1 < 2 OSTs -- skipping [29814.987002] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems == 22:51:40 (1476510700) [29820.575092] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [29821.007046] Lustre: DEBUG MARKER: FID seq 0x200007931, oid 0x1a5d ver 0x0 [29821.400430] Lustre: DEBUG MARKER: LOV seq 0x200007931, oid 0x1a5d, count: 1 [29821.804565] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:109157/0x1aa65 seq:0 [29822.492830] Lustre: Failing over lustre-OST0000 [29822.713530] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [29823.012585] Lustre: server umount lustre-OST0000 complete [29824.053553] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [29824.104961] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [29826.022887] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [29826.042621] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [29828.725159] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29828.738821] Lustre: lustre-OST0000: Connection restored to cca56237-4d15-8d3b-2ad8-30f0efd5b620 (at 10.2.4.37@tcp) [29828.753312] Lustre: Skipped 1 previous similar message [29832.188293] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [29832.190110] Lustre: lustre-OST0000: deleting orphan objects from 0x0:109159 to 0x0:109185 [29832.702925] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [29833.140872] Lustre: DEBUG MARKER: FID seq 0x200007931, oid 0x1a5f ver 0x0 [29833.525410] Lustre: DEBUG MARKER: LOV seq 0x200007931, oid 0x1a5f, count: 1 [29833.920414] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:109158/0x1aa66 seq:0 [29834.579416] Lustre: Failing over lustre-OST0000 [29835.429328] Lustre: server umount lustre-OST0000 complete [29836.429589] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [29836.472786] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null) [29838.308249] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [29838.332634] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [29841.520238] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29845.495628] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [29845.497535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:109159 to 0x0:109217 [29846.763005] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values == 22:52:12 (1476510732) [29848.002014] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 22:52:13 (1476510733) [29849.215544] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs == 22:52:15 (1476510735) [29849.596677] Lustre: DEBUG MARKER: SKIP: sanity test_27C needs >= 2 OSTs [29850.757447] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API == 22:52:16 (1476510736) [29851.142933] Lustre: DEBUG MARKER: SKIP: sanity test_27D needs >= 2 OSTs [29852.325184] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases == 22:52:18 (1476510738) [29852.714900] Lustre: DEBUG MARKER: SKIP: sanity test_27E needs >= 2 OSTs [29853.874242] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size == 22:52:19 (1476510739) [29854.920739] Lustre: Failing over lustre-OST0000 [29855.550528] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [29855.847815] Lustre: server umount lustre-OST0000 complete [29857.235653] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [29857.265632] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [29858.739766] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [29864.406431] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [29864.408334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:109223 to 0x0:109249 [29866.631249] Lustre: DEBUG MARKER: sanity test_27F: @@@@@@ FAIL: wrong stripecount [29869.990657] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============== 22:52:35 (1476510755) [29871.464059] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================== 22:52:37 (1476510757) [29871.882346] Lustre: DEBUG MARKER: first d29 [29872.278800] Lustre: DEBUG MARKER: second d29 [29872.655976] Lustre: DEBUG MARKER: done [29873.840211] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ================ 22:52:39 (1476510759) [29875.019200] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ============= 22:52:40 (1476510760) [29876.212696] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ====== 22:52:42 (1476510762) [29877.547966] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ==================================== 22:52:43 (1476510763) [29878.745984] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ========= 22:52:44 (1476510764) [29879.936874] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links =============== 22:52:45 (1476510765) [29881.147519] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =========================== 22:52:46 (1476510766) [29882.339984] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ================= 22:52:48 (1476510768) [29883.525912] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ===== 22:52:49 (1476510769) [29891.071179] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================= 22:52:56 (1476510776) [29892.291523] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================= 22:52:58 (1476510778) [29893.696927] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================= 22:52:59 (1476510779) [29895.224002] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================= 22:53:00 (1476510780) [29896.391418] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir================= 22:53:02 (1476510782) [29897.586293] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir================= 22:53:03 (1476510783) [29898.818665] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file == 22:53:04 (1476510784) [29900.001911] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename == 22:53:05 (1476510785) [29908.962274] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory == 22:53:14 (1476510794) [29909.328231] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [29910.476693] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ======================= 22:53:16 (1476510796) [29911.702084] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ======================= 22:53:17 (1476510797) [29912.947397] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir =========== 22:53:18 (1476510798) [29914.154366] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir =========== 22:53:19 (1476510799) [29915.394553] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ====== 22:53:21 (1476510801) [29916.614742] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ====== 22:53:22 (1476510802) [29917.921096] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 == 22:53:23 (1476510803) [29919.153475] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 == 22:53:24 (1476510804) [29920.351662] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file ============= 22:53:26 (1476510806) [29921.580178] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file ============= 22:53:27 (1476510807) [29922.785320] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ========== 22:53:28 (1476510808) [29923.999587] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ========== 22:53:29 (1476510809) [29925.289164] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ======== 22:53:31 (1476510811) [29926.523429] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ======== 22:53:32 (1476510812) [29927.710963] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ == 22:53:33 (1476510813) [29928.895722] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ == 22:53:34 (1476510814) [29929.290838] Lustre: DEBUG MARKER: 32p_1 [29929.678660] Lustre: DEBUG MARKER: 32p_2 [29930.084208] Lustre: DEBUG MARKER: 32p_3 [29930.467211] Lustre: DEBUG MARKER: 32p_4 [29930.863417] Lustre: DEBUG MARKER: 32p_5 [29931.232057] Lustre: DEBUG MARKER: 32p_6 [29931.655368] Lustre: DEBUG MARKER: 32p_7 [29932.039641] Lustre: DEBUG MARKER: 32p_8 [29932.432918] Lustre: DEBUG MARKER: 32p_9 [29932.822045] Lustre: DEBUG MARKER: 32p_10 [29933.968802] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) == 22:53:39 (1476510819) [29935.163452] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) == 22:53:40 (1476510820) [29936.423208] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ====== 22:53:42 (1476510822) [29937.023423] Lustre: DEBUG MARKER: 33_1 [29937.409047] Lustre: DEBUG MARKER: 33_2 [29938.796741] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) == 22:53:44 (1476510824) [29940.208430] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) == 22:53:45 (1476510825) [29941.907870] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes == 22:53:47 (1476510827) [29943.557209] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir == 22:53:49 (1476510829) [29943.974272] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [29945.149179] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode == 22:53:50 (1476510830) [29945.559054] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [29946.766243] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory == 22:53:52 (1476510832) [29947.165344] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [29948.339802] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ============= 22:53:54 (1476510834) [29949.518696] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ======= 22:53:55 (1476510835) [29950.699115] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ================ 22:53:56 (1476510836) [29951.912594] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file =============================== 22:53:57 (1476510837) [29953.100319] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ==== 22:53:58 (1476510838) [29954.318506] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ========= 22:54:00 (1476510840) [29955.523096] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ================================= 22:54:01 (1476510841) [29956.760599] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 22:54:02 (1476510842) [29960.649845] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ======= 22:54:06 (1476510846) [29961.882364] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ===================== 22:54:07 (1476510847) [29963.074689] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ====================== 22:54:08 (1476510848) [29964.250550] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ============ 22:54:10 (1476510850) [29965.706320] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ============= 22:54:11 (1476510851) [29966.971238] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ====== 22:54:12 (1476510852) [29968.176060] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ============ 22:54:13 (1476510853) [29970.392589] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ======================= 22:54:16 (1476510856) [30014.676023] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ============ 22:55:00 (1476510900) [30016.950720] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory == 22:55:02 (1476510902) [30017.341536] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [30018.498398] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ===== 22:55:04 (1476510904) [30019.661946] Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create ============================= 22:55:05 (1476510905) [30022.910895] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ======== 22:55:08 (1476510908) [30025.354092] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ============================= 22:55:11 (1476510911) [30028.642393] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ================================ 22:55:14 (1476510914) [30029.864411] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ========================== 22:55:15 (1476510915) [30031.087810] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =================== 22:55:16 (1476510916) [30034.268786] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ================================= 22:55:20 (1476510920) [30037.520163] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat =============== 22:55:23 (1476510923) [30039.757675] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ================================ 22:55:25 (1476510925) [30041.967429] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ========================= 22:55:27 (1476510927) [30045.266994] Lustre: 2 MB is too small for debug buffer size, setting it to 128 MB. [30046.835176] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ========================== 22:55:32 (1476510932) [30049.233975] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ============================= 22:55:34 (1476510934) [30050.060333] Lustre: DEBUG MARKER: sanity test_39l: @@@@@@ FAIL: atime is not updated from future: 1508046934, 1476510935= 2 MDTs [30070.480135] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate =============== 22:55:56 (1476510956) [30071.671945] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ======================= 22:55:57 (1476510957) [30073.831395] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [30074.249384] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [30074.644806] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [30075.054864] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [30075.442788] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously == 22:56:01 (1476510961) [30100.800251] Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY == 22:56:26 (1476510986) [30103.008901] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY == 22:56:28 (1476510988) [30104.268859] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY == 22:56:29 (1476510989) [30105.655196] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre========================== 22:56:31 (1476510991) [30106.900289] Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe =============== 22:56:32 (1476510992) [30107.293505] Lustre: DEBUG MARKER: SKIP: sanity test_44 skipping 2-stripe test [30108.489201] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ================================= 22:56:34 (1476510994) [30109.963523] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [30110.366744] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================== 22:56:36 (1476510996) [30111.772245] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=== 22:56:37 (1476510997) [30113.041255] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=== 22:56:38 (1476510998) [30114.247268] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) == 22:56:39 (1476510999) [30115.465264] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) == 22:56:41 (1476511001) [30116.709154] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) == 22:56:42 (1476511002) [30117.947241] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent == 22:56:43 (1476511003) [30119.516984] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ================= 22:56:45 (1476511005) [30120.719906] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ==== 22:56:46 (1476511006) [30122.537058] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit == 22:56:48 (1476511008) [30204.762190] Lustre: DEBUG MARKER: == sanity test 51ba: verify nlink for many subdirectory cleanup == 22:58:10 (1476511090) [30315.845098] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d [30316.246529] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit == 23:00:01 (1476511201) [30903.254435] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ======= 23:09:48 (1476511788) [30904.854207] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ========= 23:09:50 (1476511790) [30906.122324] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ====== 23:09:51 (1476511791) [30909.942678] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ============================ 23:09:55 (1476511795) [30911.137842] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ======================== 23:09:56 (1476511796) [30912.323380] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ======================= 23:09:57 (1476511797) [30914.292469] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ======================== 23:09:59 (1476511799) [30915.471892] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ======================== 23:10:00 (1476511800) [30916.660935] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe == 23:10:01 (1476511801) [30917.339697] Lustre: DEBUG MARKER: SKIP: sanity test_56a skipping other /usr/bin/lfs getstripe --obd test [30918.502313] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name =============================== 23:10:03 (1476511803) [30919.987066] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============================== 23:10:05 (1476511805) [30921.421504] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========= 23:10:06 (1476511806) [30922.638911] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============================== 23:10:07 (1476511807) [30924.105921] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============================== 23:10:09 (1476511809) [30925.583081] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============================== 23:10:10 (1476511810) [30927.037532] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============================== 23:10:12 (1476511812) [30928.493328] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============================== 23:10:13 (1476511813) [30929.932228] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================ 23:10:15 (1476511815) [30931.537917] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ================================= 23:10:16 (1476511816) [30932.993656] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ================================= 23:10:18 (1476511818) [30934.473784] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ============================ 23:10:19 (1476511819) [30936.346080] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works == 23:10:21 (1476511821) [30937.898753] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works == 23:10:23 (1476511823) [30939.646310] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works == 23:10:24 (1476511824) [30941.302863] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ========= 23:10:26 (1476511826) [30942.981267] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works == 23:10:28 (1476511828) [30943.358631] Lustre: DEBUG MARKER: SKIP: sanity test_56w 1 < 2 OSTs -- skipping [30944.543585] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support == 23:10:29 (1476511829) [30944.936898] Lustre: DEBUG MARKER: SKIP: sanity test_56x need 2 OST, skipping test [30946.095903] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released == 23:10:31 (1476511831) [30947.724501] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error == 23:10:33 (1476511833) [30949.801514] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir == 23:10:35 (1476511835) [30950.182689] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [30951.358280] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ==== 23:10:36 (1476511836) [30953.534555] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ===== 23:10:38 (1476511838) [30958.684934] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ================ 23:10:43 (1476511843) [30959.910036] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =========== 23:10:45 (1476511845) [30969.922628] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============ 23:10:55 (1476511855) [30970.664643] Lustre: DEBUG MARKER: test_60 run 16623 - from kernel mode [30998.281687] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [30998.297170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [31005.284215] Lustre: 23263:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476511883/real 1476511883] req@ffff880802fae900 x1548202902010576/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476511890 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31005.325553] Lustre: 23263:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [31005.339996] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [31009.199513] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [31009.212425] Lustre: Skipped 8 previous similar messages [31011.364151] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476511890/real 1476511890] req@ffff8807fe654b00 x1548202902010640/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476511896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31012.187849] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========== 23:11:37 (1476511897) [31015.367225] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536200a49d78 to 0x202953620123c3da [31016.386603] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476511890/real 1476511890] req@ffff8807fe654500 x1548202902010656/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476511901 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31020.369516] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [31021.507773] Lustre: lustre-OST0000: deleting orphan objects from 0x0:111094 to 0x0:111169 [31022.737222] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full == 23:11:48 (1476511908) [31064.597235] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 23:12:29 (1476511949) [31065.767437] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created == 23:12:31 (1476511951) [31067.271581] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================== 23:12:32 (1476511952) [31068.513041] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 23:12:33 (1476511953) [31129.847617] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ===== 23:13:35 (1476512015) [31133.782336] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ======= 23:13:38 (1476512018) [31134.943693] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [31135.353507] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ == 23:13:40 (1476512020) [31136.528113] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ====================== 23:13:41 (1476512021) [31137.794679] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 == 23:13:43 (1476512023) [31139.102751] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 0 == 23:13:44 (1476512024) [31140.253542] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count == 23:13:45 (1476512025) [31141.516784] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========================= 23:13:46 (1476512026) [31142.769434] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ===== 23:13:47 (1476512027) [31143.972071] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d ============================= 23:13:49 (1476512029) [31145.207228] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ====================== 23:13:50 (1476512030) [31146.428259] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)=== 23:13:51 (1476512031) [31147.654880] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping == 23:13:52 (1476512032) [31149.755638] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping == 23:13:54 (1476512034) [31151.481717] Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping == 23:13:56 (1476512036) [31160.312904] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=== 23:14:05 (1476512045) [31162.587608] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs == 23:14:07 (1476512047) [31163.000176] Lustre: DEBUG MARKER: SKIP: sanity test_65k too few OSTs [31164.151764] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ========================== 23:14:09 (1476512049) [31165.338010] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ================= 23:14:10 (1476512050) [31168.432157] Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) [31168.839876] Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b [31169.232037] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ======== 23:14:14 (1476512054) [31170.215353] Lustre: *** cfs_fail_loc=217, val=0*** [31170.223956] Lustre: Skipped 3 previous similar messages [31171.828875] Lustre: *** cfs_fail_loc=217, val=0*** [31171.828877] Lustre: *** cfs_fail_loc=217, val=0*** [31171.846181] Lustre: Skipped 2 previous similar messages [31173.346557] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [31173.745847] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ====== 23:14:18 (1476512058) [31175.211848] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) == 23:14:20 (1476512060) [31177.174532] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) == 23:14:22 (1476512062) [31204.459941] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) == 23:14:49 (1476512089) [31205.642317] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) == 23:14:50 (1476512090) [31206.834097] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) == 23:14:52 (1476512092) [31208.010615] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [31208.425822] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation == 23:14:53 (1476512093) [31210.935179] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read == 23:14:56 (1476512096) [31211.031643] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200007931:0xa13d:0x0] object 0x0:116195 extent [0-1048575]: client csum 111c1595, server csum 111c1594 [31211.057848] LustreError: Skipped 15 previous similar messages [31212.768627] Lustre: DEBUG MARKER: set checksum type to crc32 [31214.564300] Lustre: DEBUG MARKER: set checksum type to adler [31216.282439] Lustre: DEBUG MARKER: set checksum type to adler [31217.464898] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read == 23:15:02 (1476512102) [31217.589224] LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.36@tcp inode [0x200007931:0xa13e:0x0] object 0x0:116196 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [31220.724285] Lustre: DEBUG MARKER: SKIP: sanity test_77f skipping SLOW test 77f [31221.128405] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read == 23:15:06 (1476512106) [31221.582200] Lustre: *** cfs_fail_loc=21a, val=0*** [31223.275789] Lustre: *** cfs_fail_loc=21b, val=0*** [31226.065408] Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM == 23:15:11 (1476512111) [31226.369811] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [31226.384283] Lustre: Skipped 2 previous similar messages [31227.674243] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 == 23:15:12 (1476512112) [31231.235026] Lustre: DEBUG MARKER: SKIP: sanity test_78 skipping SLOW test 78 [31231.626750] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ========================= 23:15:16 (1476512116) [31234.221253] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ====== 23:15:19 (1476512119) [31236.909492] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ================= 23:15:22 (1476512122) [31237.303087] Lustre: *** cfs_fail_loc=228, val=0*** [31237.311331] Lustre: Skipped 3 previous similar messages [31239.113491] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ========= 23:15:24 (1476512124) [31239.513499] Lustre: *** cfs_fail_loc=228, val=0*** [31241.475205] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ================================= 23:15:26 (1476512126) [31245.169327] Lustre: DEBUG MARKER: == sanity test 99a: cvs init =========================================== 23:15:30 (1476512130) [31247.317878] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ========================================= 23:15:32 (1476512132) [31249.467090] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ======================================= 23:15:34 (1476512134) [31253.622459] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ============================================ 23:15:38 (1476512138) [31254.880124] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ========================================= 23:15:40 (1476512140) [31256.494545] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ========================================= 23:15:41 (1476512141) [31258.882307] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ============= 23:15:44 (1476512144) [31260.339025] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads ================== 23:15:45 (1476512145) [31290.513981] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =================== 23:16:15 (1476512175) [31290.912990] Lustre: DEBUG MARKER: SKIP: sanity test_101b skipping stride IO stride-ahead test [31292.087484] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead =================== 23:16:17 (1476512177) [31318.711386] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled == 23:16:43 (1476512203) [31319.122231] Lustre: DEBUG MARKER: SKIP: sanity test_101d Need free space 500M, have 160M [31320.278983] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) == 23:16:45 (1476512205) [31331.700992] Lustre: DEBUG MARKER: == sanity test 101f: check read-ahead for max_read_ahead_whole_mb == 23:16:56 (1476512216) [31333.131011] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ==================================== 23:16:58 (1476512218) [31334.418051] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ============== 23:16:59 (1476512219) [31334.813938] Lustre: DEBUG MARKER: SKIP: sanity test_102b skipping 2-stripe test [31335.990174] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ============= 23:17:01 (1476512221) [31336.387861] Lustre: DEBUG MARKER: SKIP: sanity test_102c skipping 2-stripe test [31337.565899] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============= 23:17:02 (1476512222) [31337.968031] Lustre: DEBUG MARKER: SKIP: sanity test_102d skipping N-stripe test [31339.126506] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ============= 23:17:04 (1476512224) [31339.538901] Lustre: DEBUG MARKER: SKIP: sanity test_102f skipping N-stripe test [31340.688019] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block == 23:17:05 (1476512225) [31341.105093] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [31341.528473] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [31341.914443] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [31342.333163] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [31343.533359] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode == 23:17:08 (1476512228) [31344.264483] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [31345.433472] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ============== 23:17:10 (1476512230) [31346.633190] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ===== 23:17:11 (1476512231) [31347.040387] Lustre: DEBUG MARKER: SKIP: sanity test_102j skipping N-stripe test [31348.197703] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash == 23:17:13 (1476512233) [31349.495619] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ==================================== 23:17:14 (1476512234) [31350.689976] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========== 23:17:15 (1476512235) [31351.890098] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs == 23:17:17 (1476512237) [31353.818198] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission == 23:17:18 (1476512238) [31355.323669] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans == 23:17:20 (1476512240) [31356.848058] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values == 23:17:21 (1476512241) [31358.445723] Lustre: DEBUG MARKER: == sanity test 103a: acl test =========================================== 23:17:23 (1476512243) [31376.439636] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' == 23:17:41 (1476512261) [31380.544288] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [31380.559591] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [31386.262090] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [31386.274774] Lustre: Skipped 3 previous similar messages [31387.546571] Lustre: 23264:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476512265/real 1476512265] req@ffff8807fe656300 x1548202902030832/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476512272 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31387.587257] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [31387.609844] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [31387.632287] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620123c3da to 0x20295362012dc5e1 [31391.879359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:117522 to 0x0:117601 [31392.612035] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -19 [31392.627252] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [31402.268773] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [31402.290827] LustreError: 81433:0:(mgc_request.c:571:do_requeue()) failed processing log: -5 [31402.291549] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [31402.291832] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362012dc5e1 to 0x20295362012dc7bd [31405.357529] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl == 23:18:10 (1476512290) [31405.409266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:117603 to 0x0:117633 [31406.837409] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =========================== 23:18:11 (1476512291) [31406.992282] Lustre: lustre-OST0000: Client 74b4639f-8f22-8269-1e7b-e86f88780d58 (at 10.2.4.36@tcp) reconnecting [31407.440739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88007d592800.ost_server_uuid in FULL state after 0 sec [31408.682774] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================== 23:18:13 (1476512293) [31409.887969] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ========== 23:18:14 (1476512294) [31411.120265] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ========== 23:18:16 (1476512296) [31412.398925] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========== 23:18:17 (1476512297) [31413.634203] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ========== 23:18:18 (1476512298) [31424.889130] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========= 23:18:29 (1476512309) [31426.248660] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir == 23:18:31 (1476512311) [31427.503468] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG == 23:18:32 (1476512312) [31429.765587] Lustre: DEBUG MARKER: == sanity test 110: filename length checking == 23:18:34 (1476512314) [31431.013066] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [31431.403417] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ===================== 23:18:36 (1476512316) [31431.827370] Lustre: DEBUG MARKER: SKIP: sanity test_116a 1 < 2 OSTs [31432.987049] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass == 23:18:38 (1476512318) [31436.007385] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ============ 23:18:41 (1476512321) [31437.237198] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============ 23:18:42 (1476512322) [31438.788018] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============ 23:18:43 (1476512323) [31439.276036] Lustre: *** cfs_fail_loc=217, val=0*** [31441.363516] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============ 23:18:46 (1476512326) [31441.903018] LustreError: 92785:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 sleeping for 1000ms [31442.918457] LustreError: 92785:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 216 awake [31445.557409] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ============ 23:18:50 (1476512330) [31446.080911] LustreError: 92785:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 5000ms [31451.097430] LustreError: 92785:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [31453.082718] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ============ 23:18:58 (1476512338) [31454.775458] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ============ 23:18:59 (1476512339) [31456.489533] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ============ 23:19:01 (1476512341) [31457.032164] Lustre: *** cfs_fail_loc=20e, val=0*** [31460.043953] Lustre: *** cfs_fail_loc=20e, val=0*** [31460.052590] Lustre: Skipped 1 previous similar message [31467.200450] Lustre: *** cfs_fail_loc=20e, val=0*** [31467.209018] Lustre: Skipped 1 previous similar message [31468.989737] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ============ 23:19:14 (1476512354) [31477.258720] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ============ 23:19:22 (1476512362) [31477.756678] Lustre: *** cfs_fail_loc=220, val=0*** [31477.764968] Lustre: Skipped 3 previous similar messages [31479.614366] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =========== 23:19:24 (1476512364) [31489.827213] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========== 23:19:34 (1476512374) [31491.308357] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =========== 23:19:36 (1476512376) [31492.757340] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount == 23:19:37 (1476512377) [31494.465145] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount == 23:19:39 (1476512379) [31494.869229] Lustre: DEBUG MARKER: SKIP: sanity test_119b skipping 2-stripe test [31496.041785] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole == 23:19:41 (1476512381) [31497.581628] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed == 23:19:42 (1476512382) [31499.517446] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [31502.700978] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test == 23:19:47 (1476512387) [31504.639898] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test == 23:19:49 (1476512389) [31506.511545] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test == 23:19:51 (1476512391) [31508.412752] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test == 23:19:53 (1476512393) [31510.279704] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test == 23:19:55 (1476512395) [31517.302568] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test == 23:20:02 (1476512402) [31524.273389] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test == 23:20:09 (1476512409) [31565.224617] Lustre: DEBUG MARKER: == sanity test 121: read cancel race =========== 23:20:50 (1476512450) [31566.456823] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work == 23:20:51 (1476512451) [31566.852705] Lustre: DEBUG MARKER: testing on UP system. Performance may be not as good as expected. [31567.763514] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [31568.355007] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [31573.081702] Lustre: DEBUG MARKER: ls 1000 files without statahead: 3 sec [31574.175502] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [31617.950266] Lustre: DEBUG MARKER: ls 10000 files without statahead: 27 sec [31624.815214] Lustre: DEBUG MARKER: ls 10000 files with statahead: 5 sec [31625.203733] Lustre: DEBUG MARKER: ls done [31638.946139] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 13 seconds [31639.332856] Lustre: DEBUG MARKER: rm done [31640.506487] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) == 23:22:05 (1476512525) [31645.261553] Lustre: DEBUG MARKER: ls done [31647.579074] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ========================================= 23:22:12 (1476512532) [31647.978421] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [31653.007540] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88007d592800 [31653.400289] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88007d592800 [31653.797528] Lustre: DEBUG MARKER: LRU=2003 [31654.181431] Lustre: DEBUG MARKER: LIMIT=1596099 [31654.578221] Lustre: DEBUG MARKER: LVF=956225 [31654.972779] Lustre: DEBUG MARKER: OLD_LVF=1 [31655.353361] Lustre: DEBUG MARKER: Sleep 50 sec [31705.817240] Lustre: DEBUG MARKER: Dropped 752 locks in 50s [31706.203029] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [31710.077618] Lustre: DEBUG MARKER: SKIP: sanity test_124b skipping SLOW test 124b [31710.473326] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks == 23:23:15 (1476512595) [31732.422111] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs == 23:23:37 (1476512617) [31733.800010] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account == 23:23:38 (1476512618) [31735.003614] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane == 23:23:39 (1476512619) [31736.669195] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane == 23:23:41 (1476512621) [31737.916455] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's == 23:23:42 (1476512622) [31739.263118] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ========================== 23:23:44 (1476512624) [31849.522435] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) == 23:25:34 (1476512734) [31850.957231] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) == 23:25:35 (1476512735) [31851.338331] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping FIEMAP on 2-stripe file test [31852.581543] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) == 23:25:37 (1476512737) [31852.967899] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping FIEMAP on 2-stripe file [31854.319350] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) == 23:25:39 (1476512739) [31854.685499] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [31855.871266] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) == 23:25:40 (1476512740) [31856.269228] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping continuation FIEMAP test [31857.444872] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv == 23:25:42 (1476512742) [31858.714017] Lustre: DEBUG MARKER: == sanity test 131b: test append writev == 23:25:43 (1476512743) [31860.077946] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects == 23:25:44 (1476512744) [31861.294116] Lustre: DEBUG MARKER: == sanity test 131d: test short read == 23:25:46 (1476512746) [31862.510893] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole == 23:25:47 (1476512747) [31863.757075] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ========================================== 23:25:48 (1476512748) [31869.382958] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ==================================== 23:25:54 (1476512754) [31873.021296] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ========================================== 23:25:57 (1476512757) [31890.175671] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================== 23:26:15 (1476512775) [31896.249019] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =================== 23:26:21 (1476512781) [31899.420698] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 23:26:24 (1476512784) [31904.662848] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc == 23:26:29 (1476512789) [31907.336554] LustreError: 120313:0:(ldlm_resource.c:109:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [31907.351103] LustreError: 120313:0:(ldlm_resource.c:109:seq_watermark_write()) Skipped 9 previous similar messages [31907.412655] LustreError: 120345:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Can't parse user input, rc = -14 [31907.426306] LustreError: 120345:0:(ldlm_pool.c:722:lprocfs_wr_recalc_period()) Skipped 14 previous similar messages [31907.451137] LustreError: 120355:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) invalid value written [31907.464099] LustreError: 120355:0:(ldlm_resource.c:348:lprocfs_lru_size_seq_write()) Skipped 7 previous similar messages [31907.626554] LustreError: 120435:0:(genops.c:1519:obd_export_evict_by_uuid()) lustre-OST0000: can't disconnect : no exports found [31907.645222] LustreError: 120435:0:(genops.c:1519:obd_export_evict_by_uuid()) Skipped 3 previous similar messages [31907.985380] LustreError: 120564:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) srv-lustre-OST0000: invalid FID sequence width: rc = -14 [31908.005386] LustreError: 120564:0:(lproc_fid.c:180:lprocfs_server_fid_width_seq_write()) Skipped 3 previous similar messages [31908.028273] LustreError: 120570:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) cannot add nodemap config to non-existing MGS. [31908.046309] LustreError: 120570:0:(nodemap_storage.c:404:nodemap_idx_nodemap_update()) Skipped 1 previous similar message [31908.070105] LustreError: 120578:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [31908.106393] Lustre: 120595:0:(libcfs_string.c:131:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [31908.125084] Lustre: 120595:0:(libcfs_string.c:131:cfs_str2mask()) Skipped 2 previous similar messages [31908.140165] Lustre: 0 MB is too small for debug buffer size, setting it to 128 MB. [31912.538133] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [31912.553297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [31924.543097] Lustre: 23262:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476512802/real 1476512802] req@ffff8807fe6c1200 x1548202902256736/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476512809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31924.583825] Lustre: 23262:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [31924.598015] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [31928.202451] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [31928.215208] Lustre: Skipped 6 previous similar messages [31933.762092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:142330 to 0x0:143777 [31934.625171] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [31934.647987] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362012dc7bd to 0x20295362014dd6f7 [31935.074500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [31939.627379] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [31939.642708] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [31957.968933] Lustre: server umount lustre-OST0000 complete [31980.348759] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [31980.372218] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [31985.487473] Lustre: lustre-OST0000: deleting orphan objects from 0x0:142330 to 0x0:143809 [31990.598716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [31997.164642] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold == 23:28:01 (1476512881) [32014.247909] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb == 23:28:19 (1476512899) [32044.703250] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ====== 23:28:49 (1476512929) [32046.284353] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests == 23:28:51 (1476512931) [32048.324267] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ================================= 23:28:53 (1476512933) [32054.575568] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ============================== 23:28:59 (1476512939) [32055.804117] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ========================= 23:29:00 (1476512940) [32057.247081] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks == 23:29:02 (1476512942) [32058.856496] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID == 23:29:03 (1476512943) [32061.680475] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory == 23:29:06 (1476512946) [32062.379963] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [32063.540079] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments == 23:29:08 (1476512948) [32065.169775] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid == 23:29:09 (1476512949) [32067.072729] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 23:29:11 (1476512951) [32068.584020] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 23:29:13 (1476512953) [32069.950735] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests == 23:29:14 (1476512954) [32131.225511] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on == 23:30:15 (1476513015) [32133.511181] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off == 23:30:18 (1476513018) [32135.584223] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on == 23:30:20 (1476513020) [32137.701202] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off == 23:30:22 (1476513022) [32139.695559] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on == 23:30:24 (1476513024) [32148.804984] Lustre: DEBUG MARKER: SKIP: sanity test_155e max available OST size needs > 1310720 KB [32149.984767] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off == 23:30:34 (1476513034) [32152.778269] Lustre: DEBUG MARKER: SKIP: sanity test_155f max available OST size needs > 1310720 KB [32154.100322] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on == 23:30:38 (1476513038) [32156.949349] Lustre: DEBUG MARKER: SKIP: sanity test_155g max available OST size needs > 1310720 KB [32158.140636] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off == 23:30:42 (1476513042) [32160.956114] Lustre: DEBUG MARKER: SKIP: sanity test_155h max available OST size needs > 1310720 KB [32162.137575] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables == 23:30:46 (1476513046) [32163.928679] Lustre: DEBUG MARKER: Turn on read and write cache [32165.068271] Lustre: DEBUG MARKER: Write data and read it back. [32165.459138] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [32166.668717] Lustre: DEBUG MARKER: cache hits:: before: 32, after: 35 [32167.082077] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [32167.848096] Lustre: DEBUG MARKER: cache hits:: before: 35, after: 38 [32168.249086] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [32169.385337] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [32170.506128] Lustre: DEBUG MARKER: cache hits:: before: 38, after: 41 [32170.881917] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [32171.755843] Lustre: DEBUG MARKER: cache hits:: before: 41, after: 41 [32172.146260] Lustre: DEBUG MARKER: Write data and read it back. [32172.543557] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [32173.679863] Lustre: DEBUG MARKER: cache hits:: before: 41, after: 44 [32174.088725] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [32174.898384] Lustre: DEBUG MARKER: cache hits:: before: 44, after: 44 [32175.298133] Lustre: DEBUG MARKER: Turn off read and write cache [32176.413148] Lustre: DEBUG MARKER: Write data and read it back [32176.801154] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [32178.022143] Lustre: DEBUG MARKER: cache hits:: before: 44, after: 44 [32178.422377] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [32179.540189] Lustre: DEBUG MARKER: Write data and read it back [32179.933533] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [32181.139152] Lustre: DEBUG MARKER: cache hits:: before: 44, after: 44 [32181.536369] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [32182.677612] Lustre: DEBUG MARKER: cache hits:: before: 44, after: 47 [32183.866326] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity == 23:31:08 (1476513068) [32190.960127] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [32190.974665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [32197.962732] Lustre: 23268:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476513076/real 1476513076] req@ffff880804b35700 x1548202902259680/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476513083 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [32198.001198] Lustre: 23268:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [32198.014667] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [32198.821102] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [32198.833057] Lustre: Skipped 6 previous similar messages [32204.082232] Lustre: lustre-OST0000: deleting orphan objects from 0x0:145447 to 0x0:145601 [32206.050767] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog == 23:31:30 (1476513090) [32208.039677] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362014dd87f to 0x202953620156f00d [32208.780397] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event == 23:31:33 (1476513093) [32212.481283] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event == 23:31:37 (1476513097) [32212.887770] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [32213.041775] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [32214.078377] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity == 23:31:38 (1476513098) [32219.177425] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory == 23:31:43 (1476513103) [32219.565521] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [32220.727514] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags == 23:31:45 (1476513105) [32223.177307] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity == 23:31:47 (1476513107) [32224.546985] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity == 23:31:49 (1476513109) [32224.919147] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [32226.098798] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep == 23:31:50 (1476513110) [32231.758776] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock == 23:31:56 (1476513116) [32232.142083] Lustre: DEBUG MARKER: creating a 10 Mb file [32233.649954] Lustre: DEBUG MARKER: starting reads [32234.177012] Lustre: DEBUG MARKER: truncating the file [32234.577275] Lustre: DEBUG MARKER: killing dd [32234.990579] Lustre: DEBUG MARKER: removing the temporary file [32236.153029] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ======================= 23:32:00 (1476513120) [32239.372960] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ======== 23:32:04 (1476513124) [32243.647453] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc == 23:32:08 (1476513128) [32246.500162] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 23:32:11 (1476513131) [32247.038100] Lustre: Echo OBD driver; http://www.lustre.org/ [32252.467785] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT == 23:32:17 (1476513137) [32253.354665] Lustre: Echo OBD driver; http://www.lustre.org/ [32261.175317] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ========================== 23:32:25 (1476513145) [32277.450346] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ================== 23:32:42 (1476513162) [32296.791702] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========== 23:33:01 (1476513181) [32299.128847] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap == 23:33:03 (1476513183) [32300.605136] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) == 23:33:05 (1476513185) [32301.957522] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap == 23:33:06 (1476513186) [32329.904155] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap == 23:33:34 (1476513214) [32331.204118] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps == 23:33:35 (1476513215) [32332.886792] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support == 23:33:37 (1476513217) [32335.474894] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change == 23:33:40 (1476513220) [32339.269631] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file == 23:33:43 (1476513223) [32341.296647] Lustre: DEBUG MARKER: == sanity test 200: OST pools == 23:33:45 (1476513225) [32377.665553] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes =================== 23:34:22 (1476513262) [32378.934814] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ============= 23:34:23 (1476513263) [32380.218650] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ============= 23:34:24 (1476513264) [32381.430759] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size =============== 23:34:26 (1476513266) [32382.697721] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes =================== 23:34:27 (1476513267) [32383.957437] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ============= 23:34:28 (1476513268) [32385.247738] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset ============= 23:34:29 (1476513269) [32386.505972] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size =============== 23:34:31 (1476513271) [32387.773362] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats == 23:34:32 (1476513272) [32396.644710] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [32397.042395] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.1560 [32397.768402] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [32398.153095] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.5773 [32398.880520] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [32399.268924] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.10202 [32400.005260] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [32400.398798] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.511 [32401.136633] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [32401.514594] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.5019 [32402.288219] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [32402.673181] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.22711 [32403.737693] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [32404.124124] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.22262 [32405.349721] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [32405.739706] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.28328 [32406.560999] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [32406.957694] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.3369 [32407.997846] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [32408.409992] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.14266 [32409.140838] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [32409.540475] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.7836 [32427.395270] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug == 23:35:11 (1476513311) [32428.625047] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse == 23:35:13 (1476513313) [32429.901750] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open == 23:35:14 (1476513314) [32431.229671] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open == 23:35:15 (1476513315) [32438.151162] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [32438.165786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [32445.153622] Lustre: 23259:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476513323/real 1476513323] req@ffff8807fe7ed700 x1548202902263296/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476513330 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [32445.192581] Lustre: 23259:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [32445.206277] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [32455.231423] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x202953620156f00d to 0x20295362016ada1a [32457.755051] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159649 to 0x0:160321 [32458.683869] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [32460.233792] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [32477.241195] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [32482.264765] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [32487.266933] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362016ada1a to 0x20295362016adc7b [32489.793045] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159649 to 0x0:160353 [32490.880089] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [32493.299437] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly == 23:36:17 (1476513377) [32501.904962] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ============================================== 23:36:26 (1476513386) [32504.026807] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 == 23:36:28 (1476513388) [32505.425149] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 == 23:36:29 (1476513389) [32509.387923] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 == 23:36:33 (1476513393) [32510.705153] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly == 23:36:35 (1476513395) [32514.881143] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') == 23:36:39 (1476513399) [32517.092979] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================= 23:36:41 (1476513401) [32517.501963] Lustre: DEBUG MARKER: creating a 10 Mb file [32519.123814] Lustre: DEBUG MARKER: starting reads [32519.920900] Lustre: DEBUG MARKER: truncating the file [32520.329981] Lustre: DEBUG MARKER: killing dd [32520.706158] Lustre: DEBUG MARKER: removing the temporary file [32521.901308] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND == 23:36:46 (1476513406) [32523.555661] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST == 23:36:48 (1476513408) [32529.895091] Lustre: *** cfs_fail_loc=229, val=0*** [32529.903933] Lustre: Skipped 1 previous similar message [32534.921486] Lustre: *** cfs_fail_loc=229, val=0*** [32539.947881] Lustre: *** cfs_fail_loc=229, val=0*** [32539.956528] Lustre: Skipped 1 previous similar message [32557.880441] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM == 23:37:22 (1476513442) [32583.604922] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================== 23:37:48 (1476513468) [32584.910699] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure =============== 23:37:49 (1476513469) [32586.187091] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ========================= 23:37:50 (1476513470) [32587.473950] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure == 23:37:51 (1476513471) [32587.527915] Lustre: lustre-OST0000: Client 6aca511b-1abf-0129-c35f-79a38be38534 (at 10.2.4.36@tcp) reconnecting [32587.542469] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [32587.557240] Lustre: Skipped 9 previous similar messages [32589.019752] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure == 23:37:53 (1476513473) [32589.109855] Lustre: lustre-OST0000: Client 6aca511b-1abf-0129-c35f-79a38be38534 (at 10.2.4.36@tcp) reconnecting [32590.105966] LustreError: 122311:0:(ldlm_lib.c:3238:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804278e2050 x1548235944949884/t0(0) o3->6aca511b-1abf-0129-c35f-79a38be38534@10.2.4.36@tcp:-1/-1 lens 608/432 e 0 to 0 dl 1476513516 ref 1 fl Interpret:/0/0 rc 0/0 [32590.141977] Lustre: lustre-OST0000: Bulk IO read error with 6aca511b-1abf-0129-c35f-79a38be38534 (at 10.2.4.36@tcp), client will retry: rc -110 [32591.330994] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC == 23:37:55 (1476513475) [32620.499807] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe == 23:38:24 (1476513504) [32634.862930] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 == 23:38:39 (1476513519) [32648.710526] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type == 23:38:53 (1476513533) [32650.074970] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir == 23:38:54 (1476513534) [32650.426050] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [32651.660484] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM == 23:38:56 (1476513536) [32652.875762] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks == 23:38:57 (1476513537) [32707.256309] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart == 23:39:51 (1476513591) [32756.397402] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [32756.412485] LustreError: Skipped 1 previous similar message [32756.421592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [32756.445253] Lustre: Skipped 1 previous similar message [32761.399170] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [32761.421087] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362016adc7b to 0x20295362018647c9 [32763.799217] Lustre: lustre-OST0000: deleting orphan objects from 0x0:183774 to 0x0:185345 [32766.402308] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [32775.737372] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf == 23:41:00 (1476513660) [32867.206804] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files == 23:42:31 (1476513751) [32867.929764] Lustre: DEBUG MARKER: SKIP: sanity test_229 needs >= 2 OSTs [32869.107604] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory == 23:42:33 (1476513753) [32869.488444] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [32870.688388] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory == 23:42:35 (1476513755) [32871.073410] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [32872.258012] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed == 23:42:36 (1476513756) [32872.663400] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [32873.809782] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory == 23:42:38 (1476513758) [32874.214954] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [32875.395398] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files == 23:42:39 (1476513759) [32875.793971] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [32876.962474] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 23:42:41 (1476513761) [32877.360528] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [32878.545116] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT == 23:42:42 (1476513762) [32878.941789] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [32880.095702] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root == 23:42:44 (1476513764) [32880.513680] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [32881.687089] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes == 23:42:46 (1476513766) [32882.077968] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [32883.260588] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC == 23:42:47 (1476513767) [32886.217972] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer == 23:42:50 (1476513770) [32887.854468] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [32887.874334] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) Skipped 3 previous similar messages [32887.888132] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [32887.901489] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) Skipped 3 previous similar messages [32887.915029] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [32887.928471] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) Skipped 3 previous similar messages [32887.941661] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [32887.955557] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) Skipped 3 previous similar messages [32887.968462] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [32887.980763] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) Skipped 3 previous similar messages [32887.993443] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [32888.005739] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) Skipped 3 previous similar messages [32889.394599] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [32889.413678] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) Skipped 1 previous similar message [32889.427020] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [32889.440046] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) Skipped 1 previous similar message [32889.453175] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [32889.466306] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) Skipped 1 previous similar message [32889.479100] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [32889.492719] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) Skipped 1 previous similar message [32889.505247] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [32889.517283] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) Skipped 1 previous similar message [32889.529615] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [32889.541665] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) Skipped 1 previous similar message [32890.956337] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 512 < left 1036, rollback = 7 [32890.974310] Lustre: 120993:0:(osd_internal.h:1079:osd_trans_exec_op()) Skipped 1 previous similar message [32890.987168] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [32890.999755] Lustre: 120993:0:(osd_handler.c:1639:osd_trans_dump_creds()) Skipped 1 previous similar message [32891.012507] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 [32891.025309] Lustre: 120993:0:(osd_handler.c:1646:osd_trans_dump_creds()) Skipped 1 previous similar message [32891.037811] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) write: 2/1036/0, punch: 0/0/0, quota 2/2/0 [32891.051194] Lustre: 120993:0:(osd_handler.c:1656:osd_trans_dump_creds()) Skipped 1 previous similar message [32891.063505] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [32891.075333] Lustre: 120993:0:(osd_handler.c:1663:osd_trans_dump_creds()) Skipped 1 previous similar message [32891.087487] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [32891.099371] Lustre: 120993:0:(osd_handler.c:1670:osd_trans_dump_creds()) Skipped 1 previous similar message [32894.251157] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount == 23:42:58 (1476513778) [32895.777500] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds == 23:43:00 (1476513780) [32897.448151] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds == 23:43:01 (1476513781) [32898.990368] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM == 23:43:03 (1476513783) [32899.729457] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.44 too old [32900.942413] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly == 23:43:05 (1476513785) [32904.519717] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file == 23:43:08 (1476513788) [32905.923991] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls == 23:43:10 (1476513790) [32907.115823] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency == 23:43:11 (1476513791) [32908.769702] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test == 23:43:13 (1476513793) [32933.097118] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly == 23:43:37 (1476513817) [32944.046853] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly == 23:43:48 (1476513828) [32949.010988] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) == 23:43:53 (1476513833) [32949.392289] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [32950.582116] Lustre: DEBUG MARKER: == sanity test 241: bio vs dio == 23:43:54 (1476513834) [32965.979384] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio == 23:44:10 (1476513850) [32972.919572] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable == 23:44:17 (1476513857) [32974.798059] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests == 23:44:19 (1476513859) [33002.696267] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests == 23:44:47 (1476513887) [33008.534279] Lustre: DEBUG MARKER: sanity test_244: @@@@@@ FAIL: sendfile+grouplock failed [33012.445942] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs == 23:44:56 (1476513896) [33013.645611] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length == 23:44:57 (1476513897) [33014.776956] Lustre: *** cfs_fail_loc=234, val=0*** [33014.784759] Lustre: Skipped 24 previous similar messages [33015.949965] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size == 23:45:00 (1476513900) [33017.144537] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit == 23:45:01 (1476513901) [33018.356902] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly == 23:45:02 (1476513902) [33019.636814] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool == 23:45:03 (1476513903) [33022.255907] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state == 23:45:06 (1476513906) [33026.528466] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [33026.543110] LustreError: Skipped 1 previous similar message [33026.551766] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [33033.531042] Lustre: 23255:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476513911/real 1476513911] req@ffff880802faa100 x1548202902275168/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476513918 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [33033.570389] Lustre: 23255:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [33033.584219] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [33033.605643] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x20295362018647c9 to 0x2029536201ad351b [33033.625057] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [33037.007685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214383 to 0x0:214497 [33039.281238] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test == 23:45:23 (1476513923) [33039.670429] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [33040.845931] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir == 23:45:25 (1476513925) [33041.233546] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [33042.448128] Lustre: DEBUG MARKER: == sanity test 300c: chown [33042.831693] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [33043.981619] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory == 23:45:28 (1476513928) [33044.377339] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [33045.557208] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory == 23:45:29 (1476513929) [33046.267733] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [33047.423265] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory == 23:45:31 (1476513931) [33048.152238] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [33049.341218] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory == 23:45:33 (1476513933) [33050.074898] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [33051.237390] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory == 23:45:35 (1476513935) [33051.968392] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [33053.135399] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory == 23:45:37 (1476513937) [33053.880089] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [33055.041487] Lustre: DEBUG MARKER: == sanity test 300j: test large update record == 23:45:39 (1476513939) [33055.774956] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [33056.939567] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory == 23:45:41 (1476513941) [33057.676173] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [33058.874592] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout == 23:45:43 (1476513943) [33059.605959] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [33060.789557] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS == 23:45:45 (1476513945) [33062.689419] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA == 23:45:46 (1476513946) [33063.422890] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [33064.656923] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [33065.052759] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space == 23:45:49 (1476513949) [33065.450555] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [33066.613526] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory == 23:45:50 (1476513950) [33066.992745] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [33068.212932] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file == 23:45:52 (1476513952) [33068.596091] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [33069.783844] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open == 23:45:54 (1476513954) [33070.190036] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [33071.339917] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links == 23:45:55 (1476513955) [33071.732926] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [33072.913559] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link == 23:45:57 (1476513957) [33074.260329] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled == 23:45:58 (1476513958) [33075.509864] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively == 23:45:59 (1476513959) [33076.774025] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error == 23:46:01 (1476513961) [33077.970741] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea == 23:46:02 (1476513962) [33079.550556] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread == 23:46:03 (1476513963) [33100.885871] Lustre: server umount lustre-OST0000 complete [33123.569071] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [33123.592819] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache,nodelalloc [33124.914532] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214508 to 0x0:214529 [33128.544349] Lustre: DEBUG MARKER: Using TIMEOUT=20 [33135.108726] Lustre: DEBUG MARKER: == sanity test complete, duration 3545 sec == 23:46:59 (1476514019) [116859.571578] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [116859.586731] LustreError: Skipped 1 previous similar message [116859.595841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [116859.619285] Lustre: Skipped 1 previous similar message [116866.574307] Lustre: 23265:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476597703/real 1476597703] req@ffff8807fe650900 x1548202902929232/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476597710 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [116866.614409] Lustre: 23265:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [116866.628523] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [116886.079106] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [116886.091804] Lustre: Skipped 11 previous similar messages [116886.688780] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214508 to 0x0:214561 [116891.662277] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536201ad3b50 to 0x2029536201b2d363 [116901.667137] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [116918.674551] Lustre: 23256:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476597755/real 1476597755] req@ffff8807fe651500 x1548202902929712/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476597762 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [116918.715362] Lustre: 23256:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [116918.729714] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [117003.791857] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476597822/real 1476597822] req@ffff8807fe656f00 x1548202902929872/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476597847 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [117003.833876] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [117153.864660] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476597972/real 1476597975] req@ffff8807fe651b00 x1548202902930032/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476597997 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [117153.906757] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [117454.010332] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1476598297/real 1476598297] req@ffff8807fe654200 x1548202902931120/t0(0) o250->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 520/544 e 0 to 1 dl 1476598322 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [117454.051966] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 65 previous similar messages [118069.308754] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476598887/real 1476598887] req@ffff8807fe655400 x1548202902933456/t0(0) o38->lustre-MDT0000-lwp-OST0000@10.2.4.47@tcp:12/10 lens 520/544 e 0 to 1 dl 1476598912 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [118069.350801] Lustre: 23236:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 144 previous similar messages [118247.380779] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [118247.393701] Lustre: Skipped 2 previous similar messages [118247.878408] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214508 to 0x0:214593 [118254.399489] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [118254.422305] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2029536201b2d363 to 0x2d5b4f20df05e04 [118263.143437] LustreError: 16989:0:(mgc_request.c:571:do_requeue()) failed processing log: -22 [118430.582487] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Sat Oct 15 23:27:53 PDT 2016 [118433.837028] Lustre: DEBUG MARKER: Using TIMEOUT=20 [118436.286200] Lustre: DEBUG MARKER: sanity : @@@@@@ FAIL: unable to write to /mnt/lustre/d0_runas_test as UID 500. [118578.090991] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Sat Oct 15 23:30:20 PDT 2016 [118581.349976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [118583.668034] Lustre: DEBUG MARKER: sanity : @@@@@@ FAIL: unable to write to /mnt/lustre/d0_runas_test as UID 500. [118632.408396] Lustre: Failing over lustre-OST0000 [118632.634856] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [118632.911150] Lustre: server umount lustre-OST0000 complete [118719.028391] LDISKFS-fs (sdb1): file extents enabled, maximum tree depth=5 [118719.068798] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,,errors=remount-ro,no_mbcache,nodelalloc [118719.562797] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.2.4.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [118719.728922] LustreError: 24437:0:(mgc_request.c:253:do_config_log_add()) MGC10.2.4.47@tcp: failed processing log, type 4: rc = -22 [118724.565244] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [118724.579442] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [118724.592295] Lustre: Skipped 2 previous similar messages [118724.839272] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [118724.841079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214508 to 0x0:214625 [118816.352292] Lustre: lustre-OST0000: Connection restored to d1156ca3-96de-7bd9-789e-297c7e5e7b56 (at 10.2.4.36@tcp) [118926.713859] Lustre: lustre-OST0000: Connection restored to cca56237-4d15-8d3b-2ad8-30f0efd5b620 (at 10.2.4.37@tcp) [118926.728889] Lustre: Skipped 1 previous similar message [119001.955423] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Sat Oct 15 23:37:24 PDT 2016 [119005.249258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [119007.625069] Lustre: DEBUG MARKER: sanity : @@@@@@ FAIL: unable to write to /mnt/lustre/d0_runas_test as UID 500. [355179.598149] LustreError: 11-0: lustre-MDT0000-lwp-OST0000: operation obd_ping to node 10.2.4.47@tcp failed: rc = -107 [355179.613580] LustreError: Skipped 1 previous similar message [355179.623120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.2.4.47@tcp) was lost; in progress operations using this service will wait for recovery to complete [355179.647352] Lustre: Skipped 1 previous similar message [355186.600988] Lustre: 23253:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1476835907/real 1476835907] req@ffff880823b9b900 x1548202904449920/t0(0) o400->MGC10.2.4.47@tcp@10.2.4.47@tcp:26/25 lens 224/224 e 0 to 1 dl 1476835914 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [355186.641586] Lustre: 23253:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 43 previous similar messages [355186.655955] LustreError: 166-1: MGC10.2.4.47@tcp: Connection to MGS (at 10.2.4.47@tcp) was lost; in progress operations using this service will fail [355214.980589] Lustre: lustre-OST0000: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [355221.694877] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [355231.716153] Lustre: Evicted from MGS (at 10.2.4.47@tcp) after server handle changed from 0x2d5b4f20df06202 to 0x2d5b4f20df0656d [355231.736158] Lustre: MGC10.2.4.47@tcp: Connection restored to 10.2.4.47@tcp (at 10.2.4.47@tcp) [355231.749121] Lustre: Skipped 1 previous similar message [355233.876555] Lustre: lustre-OST0000: deleting orphan objects from 0x0:214508 to 0x0:214657 [355240.391416] LustreError: 24464:0:(mgc_request.c:571:do_requeue()) failed processing log: -22