2014-06-05 11:07:44 PXELINUX 4.04 0x4e03aaf7 Copyright (C) 1994-2011 H. Peter Anvin et al 2014-06-05 11:07:44 Boot options (default = *): 2014-06-05 11:07:44 linux-2.6.32-431.17.2.1chaos.ch5.2.x86_64 (*) 2014-06-05 11:07:44 freedos 2014-06-05 11:07:44 memtest86+ 2014-06-05 11:07:44 boot: ^[[08;00H^[[09;00H 2014-06-05 11:07:46 Loading vmlinuz...... 2014-06-05 11:07:46 Loading initramfs................ready. 2014-06-05 11:07:49 Initializing cgroup subsys cpuset 2014-06-05 11:07:49 Initializing cgroup subsys cpu 2014-06-05 11:07:49 Linux version 2.6.32-431.17.2.1chaos.ch5.2.x86_64 (mockbuild@builder1) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri May 16 11:55:35 PDT 2014 2014-06-05 11:07:49 Command line: initrd=initramfs console=tty0 console=ttyS0,115200n8 crashkernel=256M BOOT_IMAGE=vmlinuz BOOTIF=01-00-1e-67-3c-e2-28 2014-06-05 11:07:49 KERNEL supported cpus: 2014-06-05 11:07:49 Intel GenuineIntel 2014-06-05 11:07:49 AMD AuthenticAMD 2014-06-05 11:07:49 Centaur CentaurHauls 2014-06-05 11:07:49 BIOS-provided physical RAM map: 2014-06-05 11:07:49 BIOS-e820: 0000000000000000 - 000000000009b400 (usable) 2014-06-05 11:07:49 BIOS-e820: 000000000009b400 - 00000000000a0000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 0000000000100000 - 00000000b9080000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000b9080000 - 00000000b90fa000 (ACPI data) 2014-06-05 11:07:49 BIOS-e820: 00000000b90fa000 - 00000000bae8e000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bae8e000 - 00000000bae9f000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bae9f000 - 00000000baef1000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baef1000 - 00000000baefa000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baefa000 - 00000000baf01000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baf01000 - 00000000baf04000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baf04000 - 00000000baf3c000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baf3c000 - 00000000baf3e000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baf3e000 - 00000000baf49000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baf49000 - 00000000baf4c000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baf4c000 - 00000000baf72000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baf72000 - 00000000baf7b000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baf7b000 - 00000000baf7f000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000baf7f000 - 00000000baf83000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000baf83000 - 00000000bafbe000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bafbe000 - 00000000bafc2000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bafc2000 - 00000000bb050000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb050000 - 00000000bb057000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb057000 - 00000000bb0cc000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0cc000 - 00000000bb0ce000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0ce000 - 00000000bb0d7000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0d7000 - 00000000bb0da000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0da000 - 00000000bb0ea000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0ea000 - 00000000bb0f9000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb0f9000 - 00000000bb141000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb141000 - 00000000bb144000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb144000 - 00000000bb169000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb169000 - 00000000bb46a000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb46a000 - 00000000bb47b000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb47b000 - 00000000bb4a3000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb4a3000 - 00000000bb60f000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bb60f000 - 00000000bb61e000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bb61e000 - 00000000bd9fb000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bd9fb000 - 00000000bdbfb000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bdbfb000 - 00000000bdcdd000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000bdcdd000 - 00000000bdde7000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000bdde7000 - 00000000bde8b000 (ACPI NVS) 2014-06-05 11:07:49 BIOS-e820: 00000000bde8b000 - 00000000bde8c000 (ACPI data) 2014-06-05 11:07:49 BIOS-e820: 00000000bde8c000 - 00000000bde8d000 (ACPI NVS) 2014-06-05 11:07:49 BIOS-e820: 00000000bde8d000 - 00000000bde8e000 (ACPI data) 2014-06-05 11:07:49 BIOS-e820: 00000000bde8e000 - 00000000bdf06000 (ACPI NVS) 2014-06-05 11:07:49 BIOS-e820: 00000000bdf06000 - 00000000bdf07000 (ACPI data) 2014-06-05 11:07:49 BIOS-e820: 00000000bdf07000 - 00000000bdf09000 (ACPI NVS) 2014-06-05 11:07:49 BIOS-e820: 00000000bdf09000 - 00000000bdf24000 (ACPI data) 2014-06-05 11:07:49 BIOS-e820: 00000000bdf24000 - 00000000bdfb1000 (ACPI NVS) 2014-06-05 11:07:49 BIOS-e820: 00000000bdfb1000 - 00000000be000000 (usable) 2014-06-05 11:07:49 BIOS-e820: 00000000be000000 - 00000000d0000000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000fed19000 - 00000000fed1a000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 00000000ffa20000 - 0000000100000000 (reserved) 2014-06-05 11:07:49 BIOS-e820: 0000000100000000 - 0000002040000000 (usable) 2014-06-05 11:07:49 DMI 2.6 present. 2014-06-05 11:07:49 SMBIOS version 2.6 @ 0xF0440 2014-06-05 11:07:49 last_pfn = 0x2040000 max_arch_pfn = 0x400000000 2014-06-05 11:07:49 x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 2014-06-05 11:07:49 last_pfn = 0xbe000 max_arch_pfn = 0x400000000 2014-06-05 11:07:49 Using GB pages for direct mapping 2014-06-05 11:07:49 init_memory_mapping: 0000000000000000-00000000be000000 2014-06-05 11:07:49 init_memory_mapping: 0000000100000000-0000002040000000 2014-06-05 11:07:49 RAMDISK: 7f214000 - 7fffe460 2014-06-05 11:07:49 ACPI: RSDP 00000000000f0410 00024 (v02 INTEL ) 2014-06-05 11:07:49 ACPI: XSDT 00000000bdf21d98 000BC (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: FACP 00000000bdf21918 000F4 (v04 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-611) 2014-06-05 11:07:49 ACPI: DSDT 00000000bdf09018 163F5 (v02 INTEL S2600JF 00000002 INTL 20100331) 2014-06-05 11:07:49 ACPI: FACS 00000000bdf21f40 00040 2014-06-05 11:07:49 ACPI: APIC 00000000bdf20718 0066A (v03 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: SPMI 00000000bdf23b98 00040 (v05 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: MCFG 00000000bdf23b18 0003C (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: WDDT 00000000bdf23f18 00040 (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: SRAT 00000000bde8be18 001A8 (v03 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: SLIT 00000000bdf23e98 00030 (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: MSCT 00000000bdf22d18 00090 (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: HPET 00000000bdf23e18 00038 (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: SSDT 00000000bdf23d18 0002B (v02 INTEL S2600JF 00001000 INTL 20100331) 2014-06-05 11:07:49 ACPI: SSDT 00000000b9080018 795C4 (v02 INTEL S2600JF 00004000 INTL 20100331) 2014-06-05 11:07:49 ACPI: SPCR 00000000bdf23d98 00050 (v01 INTEL S2600JF 06222004 INTL 20090903) 2014-06-05 11:07:49 ACPI: HEST 00000000bde8df18 000A8 (v01 INTEL S2600JF 00000001 INTL 00000001) 2014-06-05 11:07:49 ACPI: BERT 00000000bdf23c98 00030 (v01 INTEL S2600JF 00000001 INTL 00000001) 2014-06-05 11:07:49 ACPI: ERST 00000000bde8dc98 00230 (v01 INTEL S2600JF 00000001 INTL 00000001) 2014-06-05 11:07:49 ACPI: EINJ 00000000bdf21798 00130 (v01 INTEL S2600JF 00000001 INTL 00000001) 2014-06-05 11:07:49 ACPI: SSDT 00000000bdf06018 00F41 (v02 INTEL S2600JF 00000002 INTL 20100331) 2014-06-05 11:07:49 ACPI: SSDT 00000000bdf06f98 00045 (v02 INTEL S2600JF 00000001 INTL 20100331) 2014-06-05 11:07:49 ACPI: SSDT 00000000bde8bc18 00181 (v02 INTEL S2600JF 00000003 INTL 20100331) 2014-06-05 11:07:49 Setting APIC routing to flat. 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 0 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 2 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 4 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 6 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 8 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 10 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 12 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 0 -> APIC 14 -> Node 0 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 32 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 34 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 36 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 38 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 40 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 42 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 44 -> Node 1 2014-06-05 11:07:49 SRAT: PXM 1 -> APIC 46 -> Node 1 2014-06-05 11:07:49 SRAT: Node 0 PXM 0 0-c0000000 2014-06-05 11:07:49 SRAT: Node 0 PXM 0 100000000-1040000000 2014-06-05 11:07:49 SRAT: Node 1 PXM 1 1040000000-2040000000 2014-06-05 11:07:49 Bootmem setup node 0 0000000000000000-0000001040000000 2014-06-05 11:07:49 NODE_DATA [0000000000009140 - 000000000003d13f] 2014-06-05 11:07:49 bootmap [0000000000100000 - 0000000000307fff] pages 208 2014-06-05 11:07:49 (9 early reservations) ==> bootmem [0000000000 - 1040000000] 2014-06-05 11:07:49 #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] 2014-06-05 11:07:49 #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] 2014-06-05 11:07:49 #2 [0001000000 - 00020209e4] TEXT DATA BSS ==> [0001000000 - 00020209e4] 2014-06-05 11:07:49 #3 [007f214000 - 007fffe460] RAMDISK ==> [007f214000 - 007fffe460] 2014-06-05 11:07:49 #4 [000009b000 - 0000100000] BIOS reserved ==> [000009b000 - 0000100000] 2014-06-05 11:07:49 #5 [0002021000 - 0002021464] BRK ==> [0002021000 - 0002021464] 2014-06-05 11:07:49 #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] 2014-06-05 11:07:49 #7 [0000001000 - 0000001030] ACPI SLIT ==> [0000001000 - 0000001030] 2014-06-05 11:07:49 #8 [0000009000 - 0000009140] MEMNODEMAP ==> [0000009000 - 0000009140] 2014-06-05 11:07:49 Bootmem setup node 1 0000001040000000-0000002040000000 2014-06-05 11:07:49 NODE_DATA [0000001040000040 - 000000104003403f] 2014-06-05 11:07:49 bootmap [0000001040035000 - 0000001040234fff] pages 200 2014-06-05 11:07:49 (9 early reservations) ==> bootmem [1040000000 - 2040000000] 2014-06-05 11:07:49 #0 [0000000000 - 0000001000] BIOS data page 2014-06-05 11:07:49 #1 [0000006000 - 0000008000] TRAMPOLINE 2014-06-05 11:07:49 #2 [0001000000 - 00020209e4] TEXT DATA BSS 2014-06-05 11:07:49 #3 [007f214000 - 007fffe460] RAMDISK 2014-06-05 11:07:49 #4 [000009b000 - 0000100000] BIOS reserved 2014-06-05 11:07:49 #5 [0002021000 - 0002021464] BRK 2014-06-05 11:07:49 #6 [0000008000 - 0000009000] PGTABLE 2014-06-05 11:07:49 #7 [0000001000 - 0000001030] ACPI SLIT 2014-06-05 11:07:49 #8 [0000009000 - 0000009140] MEMNODEMAP 2014-06-05 11:07:50 found SMP MP-table at [ffff8800000fcd20] fcd20 2014-06-05 11:07:50 Reserving 256MB of memory at 48MB for crashkernel (System RAM: 132096MB) 2014-06-05 11:07:50 Zone PFN ranges: 2014-06-05 11:07:50 DMA 0x00000001 -> 0x00001000 2014-06-05 11:07:50 DMA32 0x00001000 -> 0x00100000 2014-06-05 11:07:50 Normal 0x00100000 -> 0x02040000 2014-06-05 11:07:50 Movable zone start PFN for each node 2014-06-05 11:07:50 early_node_map[23] active PFN ranges 2014-06-05 11:07:50 0: 0x00000001 -> 0x0000009b 2014-06-05 11:07:50 0: 0x00000100 -> 0x000b9080 2014-06-05 11:07:50 0: 0x000b90fa -> 0x000bae8e 2014-06-05 11:07:50 0: 0x000bae9f -> 0x000baef1 2014-06-05 11:07:50 0: 0x000baefa -> 0x000baf01 2014-06-05 11:07:50 0: 0x000baf04 -> 0x000baf3c 2014-06-05 11:07:50 0: 0x000baf3e -> 0x000baf49 2014-06-05 11:07:50 0: 0x000baf4c -> 0x000baf72 2014-06-05 11:07:50 0: 0x000baf7b -> 0x000baf7f 2014-06-05 11:07:50 0: 0x000baf83 -> 0x000bafbe 2014-06-05 11:07:50 0: 0x000bafc2 -> 0x000bb050 2014-06-05 11:07:50 0: 0x000bb057 -> 0x000bb0cc 2014-06-05 11:07:50 0: 0x000bb0ce -> 0x000bb0d7 2014-06-05 11:07:50 0: 0x000bb0da -> 0x000bb0ea 2014-06-05 11:07:50 0: 0x000bb0f9 -> 0x000bb141 2014-06-05 11:07:50 0: 0x000bb144 -> 0x000bb169 2014-06-05 11:07:50 0: 0x000bb46a -> 0x000bb47b 2014-06-05 11:07:50 0: 0x000bb4a3 -> 0x000bb60f 2014-06-05 11:07:50 0: 0x000bb61e -> 0x000bd9fb 2014-06-05 11:07:50 0: 0x000bdbfb -> 0x000bdcdd 2014-06-05 11:07:50 0: 0x000bdfb1 -> 0x000be000 2014-06-05 11:07:50 0: 0x00100000 -> 0x01040000 2014-06-05 11:07:50 1: 0x01040000 -> 0x02040000 2014-06-05 11:07:50 ACPI: PM-Timer IO Port: 0x408 2014-06-05 11:07:50 Setting APIC routing to flat. 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled) 2014-06-05 11:07:50 ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled) 2014-06-05 11:07:50 ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) 2014-06-05 11:07:50 ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) 2014-06-05 11:07:50 IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 2014-06-05 11:07:50 ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24]) 2014-06-05 11:07:50 IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47 2014-06-05 11:07:50 ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48]) 2014-06-05 11:07:50 IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71 2014-06-05 11:07:50 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 2014-06-05 11:07:50 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) 2014-06-05 11:07:50 Using ACPI (MADT) for SMP configuration information 2014-06-05 11:07:50 ACPI: HPET id: 0x8086a701 base: 0xfed00000 2014-06-05 11:07:50 SMP: Allowing 128 CPUs, 112 hotplug CPUs 2014-06-05 11:07:50 PM: Registered nosave memory: 000000000009b000 - 000000000009c000 2014-06-05 11:07:50 PM: Registered nosave memory: 000000000009c000 - 00000000000a0000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000000e0000 - 0000000000100000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000b9080000 - 00000000b90fa000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bae8e000 - 00000000bae9f000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baef1000 - 00000000baefa000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baf01000 - 00000000baf04000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baf3c000 - 00000000baf3e000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baf49000 - 00000000baf4c000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baf72000 - 00000000baf7b000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000baf7f000 - 00000000baf83000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bafbe000 - 00000000bafc2000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb050000 - 00000000bb057000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb0cc000 - 00000000bb0ce000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb0d7000 - 00000000bb0da000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb0ea000 - 00000000bb0f9000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb141000 - 00000000bb144000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb169000 - 00000000bb46a000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb47b000 - 00000000bb4a3000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bb60f000 - 00000000bb61e000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bd9fb000 - 00000000bdbfb000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bdcdd000 - 00000000bdde7000 2014-06-05 11:07:50 PM: Registered nosave memory: 00000000bdde7000 - 00000000bde8b000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bde8b000 - 00000000bde8c000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bde8c000 - 00000000bde8d000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bde8d000 - 00000000bde8e000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bde8e000 - 00000000bdf06000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bdf06000 - 00000000bdf07000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bdf07000 - 00000000bdf09000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bdf09000 - 00000000bdf24000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000bdf24000 - 00000000bdfb1000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000be000000 - 00000000d0000000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000d0000000 - 00000000fec00000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fec01000 - 00000000fed19000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fed19000 - 00000000fed1a000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fed1a000 - 00000000fed1c000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fed1c000 - 00000000fed20000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fed20000 - 00000000fee00000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000fee01000 - 00000000ffa20000 2014-06-05 11:07:51 PM: Registered nosave memory: 00000000ffa20000 - 0000000100000000 2014-06-05 11:07:51 Allocating PCI resources starting at d0000000 (gap: d0000000:2ec00000) 2014-06-05 11:07:51 Booting paravirtualized kernel on bare hardware 2014-06-05 11:07:51 NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2 2014-06-05 11:07:51 PERCPU: Embedded 31 pages/cpu @ffff880060600000 s94872 r8192 d23912 u131072 2014-06-05 11:07:51 pcpu-alloc: s94872 r8192 d23912 u131072 alloc=1*2097152 2014-06-05 11:07:51 pcpu-alloc: [0] 000 001 002 003 004 005 006 007 016 018 020 022 024 026 028 030 2014-06-05 11:07:51 pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 2014-06-05 11:07:51 pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 2014-06-05 11:07:51 pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 2014-06-05 11:07:51 pcpu-alloc: [1] 008 009 010 011 012 013 014 015 017 019 021 023 025 027 029 031 2014-06-05 11:07:51 pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 2014-06-05 11:07:51 pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 2014-06-05 11:07:51 pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 2014-06-05 11:07:51 Built 2 zonelists in Zone order, mobility grouping on. Total pages: 33081434 2014-06-05 11:07:51 Policy zone: Normal 2014-06-05 11:07:51 Kernel command line: initrd=initramfs console=tty0 console=ttyS0,115200n8 crashkernel=256M BOOT_IMAGE=vmlinuz BOOTIF=01-00-1e-67-3c-e2-28 2014-06-05 11:07:51 PID hash table entries: 4096 (order: 3, 32768 bytes) 2014-06-05 11:07:51 xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 2014-06-05 11:07:51 Checking aperture... 2014-06-05 11:07:51 No AGP bridge found 2014-06-05 11:07:51 PCI-DMA: Using software bounce buffering for IO (SWIOTLB) 2014-06-05 11:07:51 Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 2014-06-05 11:07:51 software IO TLB at phys 0x20000000 - 0x24000000 2014-06-05 11:07:51 Memory: 131964704k/135266304k available (5337k kernel code, 1090804k absent, 2210796k reserved, 7001k data, 1280k init) 2014-06-05 11:07:51 Hierarchical RCU implementation. 2014-06-05 11:07:51 NR_IRQS:33024 nr_irqs:2248 2014-06-05 11:07:51 Extended CMOS year: 2000 2014-06-05 11:07:51 Console: colour VGA+ 80x25 2014-06-05 11:07:51 console [tty0] enabled 2014-06-05 11:07:51 console [ttyS0] enabled 2014-06-05 11:07:51 allocated 536870912 bytes of page_cgroup 2014-06-05 11:07:51 please try 'cgroup_disable=memory' option if you don't want memory cgroups 2014-06-05 11:07:51 TSC: cpu family 6 model 45, tsc initial value = 3d70f98d78 2014-06-05 11:07:51 Fast TSC calibration using PIT 2014-06-05 11:07:51 Detected 2593.607 MHz processor. 2014-06-05 11:07:51 Calibrating delay loop (skipped), value calculated using timer frequency.. 5187.21 BogoMIPS (lpj=2593607) 2014-06-05 11:07:51 pid_max: default: 131072 minimum: 1024 2014-06-05 11:07:51 Security Framework initialized 2014-06-05 11:07:51 SELinux: Initializing. 2014-06-05 11:07:51 Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) 2014-06-05 11:07:51 Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) 2014-06-05 11:07:51 Mount-cache hash table entries: 256 2014-06-05 11:07:51 Initializing cgroup subsys ns 2014-06-05 11:07:51 Initializing cgroup subsys cpuacct 2014-06-05 11:07:51 Initializing cgroup subsys memory 2014-06-05 11:07:51 Initializing cgroup subsys devices 2014-06-05 11:07:51 Initializing cgroup subsys freezer 2014-06-05 11:07:51 Initializing cgroup subsys net_cls 2014-06-05 11:07:51 Initializing cgroup subsys blkio 2014-06-05 11:07:51 Initializing cgroup subsys perf_event 2014-06-05 11:07:51 Initializing cgroup subsys net_prio 2014-06-05 11:07:51 CPU: Physical Processor ID: 0 2014-06-05 11:07:51 CPU: Processor Core ID: 0 2014-06-05 11:07:51 mce: CPU supports 20 MCE banks 2014-06-05 11:07:51 CPU0: Thermal monitoring enabled (TM1) 2014-06-05 11:07:51 using mwait in idle threads. 2014-06-05 11:07:51 ACPI: Core revision 20090903 2014-06-05 11:07:51 ftrace: converting mcount calls to 0f 1f 44 00 00 2014-06-05 11:07:51 ftrace: allocating 21824 entries in 86 pages 2014-06-05 11:07:51 x2apic not enabled, IRQ remapping init failed 2014-06-05 11:07:51 APIC routing finalized to physical flat. 2014-06-05 11:07:51 ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 2014-06-05 11:07:51 CPU0: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz stepping 06 2014-06-05 11:07:51 Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, Intel PMU driver. 2014-06-05 11:07:51 PEBS disabled due to CPU errata. 2014-06-05 11:07:51 ... version: 3 2014-06-05 11:07:51 ... bit width: 48 2014-06-05 11:07:51 ... generic registers: 8 2014-06-05 11:07:51 ... value mask: 0000ffffffffffff 2014-06-05 11:07:51 ... max period: 000000007fffffff 2014-06-05 11:07:51 ... fixed-purpose events: 3 2014-06-05 11:07:51 ... event mask: 00000007000000ff 2014-06-05 11:07:51 NMI watchdog enabled, takes one hw-pmu counter. 2014-06-05 11:07:51 Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 Ok. 2014-06-05 11:07:52 Booting Node 1, Processors #8 #9 #10 #11 #12 #13 #14 #15 2014-06-05 11:07:52 Brought up 16 CPUs 2014-06-05 11:07:52 Total of 16 processors activated (82991.90 BogoMIPS). 2014-06-05 11:07:52 devtmpfs: initialized 2014-06-05 11:07:52 PM: Registering ACPI NVS region at bdde7000 (671744 bytes) 2014-06-05 11:07:52 PM: Registering ACPI NVS region at bde8c000 (4096 bytes) 2014-06-05 11:07:52 PM: Registering ACPI NVS region at bde8e000 (491520 bytes) 2014-06-05 11:07:52 PM: Registering ACPI NVS region at bdf07000 (8192 bytes) 2014-06-05 11:07:52 PM: Registering ACPI NVS region at bdf24000 (577536 bytes) 2014-06-05 11:07:52 regulator: core version 0.5 2014-06-05 11:07:52 NET: Registered protocol family 16 2014-06-05 11:07:52 ACPI FADT declares the system doesn't support PCIe ASPM, so disable it 2014-06-05 11:07:52 ACPI: bus type pci registered 2014-06-05 11:07:52 PCI: MCFG configuration 0: base c0000000 segment 0 buses 0 - 255 2014-06-05 11:07:52 PCI: MCFG area at c0000000 reserved in E820 2014-06-05 11:07:52 PCI: Using MMCONFIG at c0000000 - cfffffff 2014-06-05 11:07:52 PCI: Using configuration type 1 for base access 2014-06-05 11:07:52 bio: create slab <bio-0> at 0 2014-06-05 11:07:52 ACPI: Executed 1 blocks of module-level executable AML code 2014-06-05 11:07:52 ACPI: BIOS _OSI(Linux) query ignored 2014-06-05 11:07:52 ACPI: Interpreter enabled 2014-06-05 11:07:52 ACPI: (supports S0 S1 S5) 2014-06-05 11:07:52 ACPI: Using IOAPIC for interrupt routing 2014-06-05 11:07:52 ACPI: No dock devices found. 2014-06-05 11:07:52 HEST: Table parsing has been initialized. 2014-06-05 11:07:52 PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug 2014-06-05 11:07:52 ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [io 0x0000-0xbfff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000c0000-0x000c3fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000c7fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000c8000-0x000cbfff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000cc000-0x000cffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000d0000-0x000d3fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000d4000-0x000d7fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000d8000-0x000dbfff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000dc000-0x000dffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000e0000-0x000e3fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000e4000-0x000e7fff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000e8000-0x000ebfff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000ec000-0x000effff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x000f0000-0x000fffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0xd0000000-0xebffffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: host bridge window [mem 0x3c0000000000-0x3c007fffffff] 2014-06-05 11:07:52 pci_root PNP0A08:00: ignoring host bridge window [mem 0x000cc000-0x000cffff] (conflicts with Adapter ROM [mem 0x000c8000-0x000ce5ff]) 2014-06-05 11:07:52 pci_root PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff] (conflicts with Adapter ROM [mem 0x000cf800-0x000d07ff]) 2014-06-05 11:07:52 PCI host bridge to bus 0000:00 2014-06-05 11:07:52 pci_bus 0000:00: root bus resource [io 0x0000-0xbfff] 2014-06-05 11:07:52 pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] 2014-06-05 11:07:52 pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff] 2014-06-05 11:07:52 pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff] 2014-06-05 11:07:52 pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0xd0000000-0xebffffff] 2014-06-05 11:07:53 pci_bus 0000:00: root bus resource [mem 0x3c0000000000-0x3c007fffffff] 2014-06-05 11:07:53 pci 0000:00:00.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:00.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:01.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:01.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:02.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:02.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:03.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:03.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:03.2: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:03.2: PME# disabled 2014-06-05 11:07:53 pci 0000:00:11.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:11.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:16.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:16.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:16.1: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:16.1: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:1a.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:1c.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:1c.7: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:00:1d.0: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1f.2: PME# supported from D3hot 2014-06-05 11:07:53 pci 0000:00:1f.2: PME# disabled 2014-06-05 11:07:53 pci 0000:00:01.0: PCI bridge to [bus 01-01] 2014-06-05 11:07:53 pci 0000:00:02.0: PCI bridge to [bus 02-02] 2014-06-05 11:07:53 pci 0000:00:03.0: PCI bridge to [bus 03-03] 2014-06-05 11:07:53 pci 0000:00:03.2: PCI bridge to [bus 04-04] 2014-06-05 11:07:53 pci 0000:00:11.0: PCI bridge to [bus 05-05] 2014-06-05 11:07:53 pci 0000:06:00.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:06:00.0: PME# disabled 2014-06-05 11:07:53 pci 0000:06:00.1: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:06:00.1: PME# disabled 2014-06-05 11:07:53 pci 0000:00:1c.0: PCI bridge to [bus 06-07] 2014-06-05 11:07:53 pci 0000:00:1c.7: PCI bridge to [bus 08-08] 2014-06-05 11:07:53 pci 0000:00:1e.0: PCI bridge to [bus 09-09] (subtractive decode) 2014-06-05 11:07:53 pci0000:00: Requesting ACPI _OSC control (0x1d) 2014-06-05 11:07:53 pci0000:00: ACPI _OSC control (0x15) granted 2014-06-05 11:07:53 ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) 2014-06-05 11:07:53 pci_root PNP0A08:01: host bridge window [io 0x03b0-0x03df] 2014-06-05 11:07:53 pci_root PNP0A08:01: host bridge window [io 0xc000-0xffff] 2014-06-05 11:07:53 pci_root PNP0A08:01: host bridge window [mem 0x000a0000-0x000bffff] 2014-06-05 11:07:53 pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff] 2014-06-05 11:07:53 pci_root PNP0A08:01: host bridge window [mem 0x3c0080000000-0x3c00ffffffff] 2014-06-05 11:07:53 PCI host bridge to bus 0000:80 2014-06-05 11:07:53 pci_bus 0000:80: root bus resource [io 0x03b0-0x03df] 2014-06-05 11:07:53 pci_bus 0000:80: root bus resource [io 0xc000-0xffff] 2014-06-05 11:07:53 pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff] 2014-06-05 11:07:53 pci_bus 0000:80: root bus resource [mem 0xec000000-0xfbffffff] 2014-06-05 11:07:53 pci_bus 0000:80: root bus resource [mem 0x3c0080000000-0x3c00ffffffff] 2014-06-05 11:07:53 pci 0000:80:03.0: PME# supported from D0 D3hot D3cold 2014-06-05 11:07:53 pci 0000:80:03.0: PME# disabled 2014-06-05 11:07:53 pci 0000:80:03.0: PCI bridge to [bus 81-81] 2014-06-05 11:07:53 pci0000:80: Requesting ACPI _OSC control (0x1d) 2014-06-05 11:07:53 pci0000:80: ACPI _OSC control (0x15) granted 2014-06-05 11:07:53 ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f]) 2014-06-05 11:07:53 PCI host bridge to bus 0000:7f 2014-06-05 11:07:53 pci0000:7f: Requesting ACPI _OSC control (0x1d) 2014-06-05 11:07:53 Unable to assume _OSC PCIe control. Disabling ASPM 2014-06-05 11:07:53 ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff]) 2014-06-05 11:07:53 PCI host bridge to bus 0000:ff 2014-06-05 11:07:53 pci0000:ff: Requesting ACPI _OSC control (0x1d) 2014-06-05 11:07:53 Unable to assume _OSC PCIe control. Disabling ASPM 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15) 2014-06-05 11:07:53 ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15) 2014-06-05 11:07:53 vgaarb: device added: PCI:0000:08:00.0,decodes=io+mem,owns=io+mem,locks=none 2014-06-05 11:07:53 vgaarb: loaded 2014-06-05 11:07:53 vgaarb: bridge control possible 0000:08:00.0 2014-06-05 11:07:53 SCSI subsystem initialized 2014-06-05 11:07:53 usbcore: registered new interface driver usbfs 2014-06-05 11:07:53 usbcore: registered new interface driver hub 2014-06-05 11:07:53 usbcore: registered new device driver usb 2014-06-05 11:07:53 PCI: Using ACPI for IRQ routing 2014-06-05 11:07:53 NetLabel: Initializing 2014-06-05 11:07:53 NetLabel: domain hash size = 128 2014-06-05 11:07:53 NetLabel: protocols = UNLABELED CIPSOv4 2014-06-05 11:07:53 NetLabel: unlabeled traffic allowed by default 2014-06-05 11:07:53 HPET: 8 timers in total, 5 timers will be used for per-cpu timer 2014-06-05 11:07:53 hpet0: at MMIO 0xfed00000, IRQs 2, 8, 72, 73, 74, 75, 76, 0 2014-06-05 11:07:53 hpet0: 8 comparators, 64-bit 14.318180 MHz counter 2014-06-05 11:07:53 Switching to clocksource hpet 2014-06-05 11:07:53 pnp: PnP ACPI init 2014-06-05 11:07:53 ACPI: bus type pnp registered 2014-06-05 11:07:53 pnp: PnP ACPI: found 15 devices 2014-06-05 11:07:53 ACPI: ACPI bus type pnp unregistered 2014-06-05 11:07:53 system 00:05: [io 0x0680-0x069f] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0xffff] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0xffff] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0xffff] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0400-0x0453] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0458-0x047f] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0500-0x057f] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0600-0x061f] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0ca2-0x0ca5] has been reserved 2014-06-05 11:07:53 system 00:05: [io 0x0cf9] could not be reserved 2014-06-05 11:07:53 system 00:07: [io 0x0454-0x0457] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xebfff000-0xebffffff] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xc0000000-0xcfffffff] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xff000000-0xffffffff] could not be reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xfec00000-0xfecfffff] could not be reserved 2014-06-05 11:07:53 system 00:0a: [mem 0xd0f70000-0xd0f70fff] has been reserved 2014-06-05 11:07:53 system 00:0c: [mem 0x00000000-0x0009cfff] could not be reserved 2014-06-05 11:07:53 pci 0000:00:01.0: PCI bridge to [bus 01-01] 2014-06-05 11:07:53 pci 0000:00:01.0: PCI bridge to [bus 01-01] 2014-06-05 11:07:53 pci 0000:00:01.0: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:00:01.0: bridge window [mem 0xd0e00000-0xd0efffff] 2014-06-05 11:07:53 pci 0000:00:01.0: bridge window [mem 0xeb000000-0xeb7fffff 64bit pref] 2014-06-05 11:07:53 pci 0000:00:02.0: PCI bridge to [bus 02-02] 2014-06-05 11:07:53 pci 0000:00:02.0: PCI bridge to [bus 02-02] 2014-06-05 11:07:53 pci 0000:00:02.0: bridge window [io 0x3000-0x3fff] 2014-06-05 11:07:53 pci 0000:00:02.0: bridge window [mem 0xd0b00000-0xd0cfffff] 2014-06-05 11:07:53 pci 0000:00:02.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:00:03.0: PCI bridge to [bus 03-03] 2014-06-05 11:07:53 pci 0000:00:03.0: PCI bridge to [bus 03-03] 2014-06-05 11:07:53 pci 0000:00:03.0: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:00:03.0: bridge window [mem disabled] 2014-06-05 11:07:53 pci 0000:00:03.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:00:03.2: PCI bridge to [bus 04-04] 2014-06-05 11:07:53 pci 0000:00:03.2: PCI bridge to [bus 04-04] 2014-06-05 11:07:53 pci 0000:00:03.2: bridge window [io 0x2000-0x2fff] 2014-06-05 11:07:53 pci 0000:00:03.2: bridge window [mem 0xd0900000-0xd0afffff] 2014-06-05 11:07:53 pci 0000:00:03.2: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:00:11.0: PCI bridge to [bus 05-05] 2014-06-05 11:07:53 pci 0000:00:11.0: PCI bridge to [bus 05-05] 2014-06-05 11:07:53 pci 0000:00:11.0: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:00:11.0: bridge window [mem disabled] 2014-06-05 11:07:53 pci 0000:00:11.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:06:00.0: BAR 7: assigned [mem 0xd0d54000-0xd0d73fff] 2014-06-05 11:07:53 pci 0000:06:00.0: BAR 7: set to [mem 0xd0d54000-0xd0d73fff] (PCI address [0xd0d54000-0xd0d73fff] 2014-06-05 11:07:53 pci 0000:06:00.0: BAR 10: assigned [mem 0xd0d74000-0xd0d93fff] 2014-06-05 11:07:53 pci 0000:06:00.0: BAR 10: set to [mem 0xd0d74000-0xd0d93fff] (PCI address [0xd0d74000-0xd0d93fff] 2014-06-05 11:07:53 pci 0000:06:00.1: BAR 7: assigned [mem 0xd0d94000-0xd0db3fff] 2014-06-05 11:07:53 pci 0000:06:00.1: BAR 7: set to [mem 0xd0d94000-0xd0db3fff] (PCI address [0xd0d94000-0xd0db3fff] 2014-06-05 11:07:53 pci 0000:06:00.1: BAR 10: assigned [mem 0xd0db4000-0xd0dd3fff] 2014-06-05 11:07:53 pci 0000:06:00.1: BAR 10: set to [mem 0xd0db4000-0xd0dd3fff] (PCI address [0xd0db4000-0xd0dd3fff] 2014-06-05 11:07:53 pci 0000:00:1c.0: PCI bridge to [bus 06-07] 2014-06-05 11:07:53 pci 0000:00:1c.0: PCI bridge to [bus 06-07] 2014-06-05 11:07:53 pci 0000:00:1c.0: bridge window [io 0x1000-0x1fff] 2014-06-05 11:07:53 pci 0000:00:1c.0: bridge window [mem 0xd0d00000-0xd0dfffff] 2014-06-05 11:07:53 pci 0000:00:1c.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:00:1c.7: PCI bridge to [bus 08-08] 2014-06-05 11:07:53 pci 0000:00:1c.7: PCI bridge to [bus 08-08] 2014-06-05 11:07:53 pci 0000:00:1c.7: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] 2014-06-05 11:07:53 pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] 2014-06-05 11:07:53 pci 0000:00:1e.0: PCI bridge to [bus 09-09] 2014-06-05 11:07:53 pci 0000:00:1e.0: PCI bridge to [bus 09-09] 2014-06-05 11:07:53 pci 0000:00:1e.0: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:00:1e.0: bridge window [mem disabled] 2014-06-05 11:07:53 pci 0000:00:1e.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:80:03.0: PCI bridge to [bus 81-81] 2014-06-05 11:07:53 pci 0000:80:03.0: PCI bridge to [bus 81-81] 2014-06-05 11:07:53 pci 0000:80:03.0: bridge window [io disabled] 2014-06-05 11:07:53 pci 0000:80:03.0: bridge window [mem disabled] 2014-06-05 11:07:53 pci 0000:80:03.0: bridge window [mem pref disabled] 2014-06-05 11:07:53 pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47 2014-06-05 11:07:53 pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47 2014-06-05 11:07:53 pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2014-06-05 11:07:54 pci 0000:00:03.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2014-06-05 11:07:54 pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2014-06-05 11:07:54 pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2014-06-05 11:07:54 pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19 2014-06-05 11:07:54 pci 0000:80:03.0: PCI INT A -> GSI 71 (level, low) -> IRQ 71 2014-06-05 11:07:54 NET: Registered protocol family 2 2014-06-05 11:07:54 IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) 2014-06-05 11:07:54 TCP established hash table entries: 524288 (order: 11, 8388608 bytes) 2014-06-05 11:07:54 TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) 2014-06-05 11:07:54 TCP: Hash tables configured (established 524288 bind 65536) 2014-06-05 11:07:54 TCP reno registered 2014-06-05 11:07:54 NET: Registered protocol family 1 2014-06-05 11:07:54 pci 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22 2014-06-05 11:07:54 pci 0000:00:1a.0: PCI INT B disabled 2014-06-05 11:07:54 pci 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20 2014-06-05 11:07:54 pci 0000:00:1d.0: PCI INT A disabled 2014-06-05 11:07:54 Unpacking initramfs... 2014-06-05 11:07:54 Freeing initrd memory: 14249k freed 2014-06-05 11:07:54 audit: initializing netlink socket (disabled) 2014-06-05 11:07:54 type=2000 audit(1401991670.622:1): initialized 2014-06-05 11:07:54 HugeTLB registered 2 MB page size, pre-allocated 0 pages 2014-06-05 11:07:54 VFS: Disk quotas dquot_6.5.2 2014-06-05 11:07:54 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) 2014-06-05 11:07:54 msgmni has been set to 32768 2014-06-05 11:07:54 alg: No test for stdrng (krng) 2014-06-05 11:07:54 ksign: Installing public key data 2014-06-05 11:07:54 Loading keyring 2014-06-05 11:07:54 - Added public key F5D54BFDF0EBE5F4 2014-06-05 11:07:54 - User ID: Red Hat, Inc. (Kernel Module GPG key) 2014-06-05 11:07:54 - Added public key D4A26C9CCD09BEDA 2014-06-05 11:07:54 - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com> 2014-06-05 11:07:54 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) 2014-06-05 11:07:54 io scheduler noop registered 2014-06-05 11:07:54 io scheduler anticipatory registered 2014-06-05 11:07:54 io scheduler deadline registered 2014-06-05 11:07:54 io scheduler cfq registered (default) 2014-06-05 11:07:54 pci_hotplug: PCI Hot Plug PCI Core version: 0.5 2014-06-05 11:07:54 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 2014-06-05 11:07:54 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 2014-06-05 11:07:54 acpiphp: Slot [2] registered 2014-06-05 11:07:54 acpiphp: Slot [2-1] registered 2014-06-05 11:07:54 acpiphp: Slot [2-2] registered 2014-06-05 11:07:54 ipmi message handler version 39.2 2014-06-05 11:07:54 IPMI System Interface driver. 2014-06-05 11:07:54 ipmi_si: probing via SMBIOS 2014-06-05 11:07:54 ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 2014-06-05 11:07:54 ipmi_si: Adding SMBIOS-specified kcs state machine 2014-06-05 11:07:54 ipmi_si: probing via SPMI 2014-06-05 11:07:54 ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0 2014-06-05 11:07:54 ipmi_si: Adding SPMI-specified kcs state machine duplicate interface 2014-06-05 11:07:54 ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 2014-06-05 11:07:55 ipmi_si ipmi_si.0: Found new BMC (man_id: 0x000157, prod_id: 0x004d, dev_id: 0x21) 2014-06-05 11:07:55 ipmi_si ipmi_si.0: IPMI kcs interface initialized 2014-06-05 11:07:55 input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 2014-06-05 11:07:55 ACPI: Power Button [PWRF] 2014-06-05 11:07:55 [Firmware Bug]: No valid trip found 2014-06-05 11:07:55 ERST: Error Record Serialization Table (ERST) support is initialized. 2014-06-05 11:07:55 GHES: ghes_init() 2014-06-05 11:07:55 GHES: ghes_probe(0xffffc9000c098f54 [0 0xbdd8d018 64], type 3) 2014-06-05 11:07:55 GHES: ghes_new(0xffffc9000c098f54 [0 0xbdd8d018 64]) 2014-06-05 11:07:55 GHES: ghes_new() exiting w/ 0xffffc9000c098f54 2014-06-05 11:07:55 GHES: ghes_probe() exiting w/ 0 2014-06-05 11:07:55 GHES: ghes_probe(0xffffc9000c098f94 [0 0xbdd8e020 64], type 4) 2014-06-05 11:07:55 GHES: ghes_new(0xffffc9000c098f94 [0 0xbdd8e020 64]) 2014-06-05 11:07:55 GHES: ghes_new() exiting w/ 0xffffc9000c098f94 2014-06-05 11:07:55 GHES: ghes_probe() exiting w/ 0 2014-06-05 11:07:55 GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. 2014-06-05 11:07:55 GHES: ghes_init() exiting w/ 0 2014-06-05 11:07:55 Non-volatile memory driver v1.3 2014-06-05 11:07:55 Linux agpgart interface v0.103 2014-06-05 11:07:55 crash memory driver: version 1.1 2014-06-05 11:07:55 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled 2014-06-05 11:07:55 ~?Refined TSC clocksource calibration: 2593.749 MHz. 2014-06-05 11:07:55 Switching to clocksource tsc 2014-06-05 11:07:55 ~?serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 2014-06-05 11:07:55 serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 2014-06-05 11:07:55 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 2014-06-05 11:07:55 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 2014-06-05 11:07:55 loop: module loaded 2014-06-05 11:07:55 input: Macintosh mouse button emulation as /devices/virtual/input/input1 2014-06-05 11:07:55 Fixed MDIO Bus: probed 2014-06-05 11:07:55 ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: PCI INT B -> GSI 22 (level, low) -> IRQ 22 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: EHCI Host Controller 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: debug port 2 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: irq 22, io mem 0xd0f20000 2014-06-05 11:07:55 ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00 2014-06-05 11:07:55 usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 2014-06-05 11:07:55 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2014-06-05 11:07:55 usb usb1: Product: EHCI Host Controller 2014-06-05 11:07:55 usb usb1: Manufacturer: Linux 2.6.32-431.17.2.1chaos.ch5.2.x86_64 ehci_hcd 2014-06-05 11:07:55 usb usb1: SerialNumber: 0000:00:1a.0 2014-06-05 11:07:55 usb usb1: configuration #1 chosen from 1 choice 2014-06-05 11:07:55 hub 1-0:1.0: USB hub found 2014-06-05 11:07:55 hub 1-0:1.0: 2 ports detected 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: EHCI Host Controller 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: debug port 2 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: irq 20, io mem 0xd0f10000 2014-06-05 11:07:55 ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00 2014-06-05 11:07:55 usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 2014-06-05 11:07:55 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 2014-06-05 11:07:55 usb usb2: Product: EHCI Host Controller 2014-06-05 11:07:55 usb usb2: Manufacturer: Linux 2.6.32-431.17.2.1chaos.ch5.2.x86_64 ehci_hcd 2014-06-05 11:07:55 usb usb2: SerialNumber: 0000:00:1d.0 2014-06-05 11:07:55 usb usb2: configuration #1 chosen from 1 choice 2014-06-05 11:07:55 hub 2-0:1.0: USB hub found 2014-06-05 11:07:55 hub 2-0:1.0: 2 ports detected 2014-06-05 11:07:55 ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver 2014-06-05 11:07:55 uhci_hcd: USB Universal Host Controller Interface driver 2014-06-05 11:07:55 PNP: No PS/2 controller found. Probing ports directly. 2014-06-05 11:07:56 usb 1-1: new high speed USB device number 2 using ehci_hcd 2014-06-05 11:07:56 usb 1-1: New USB device found, idVendor=8087, idProduct=0024 2014-06-05 11:07:56 usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 2014-06-05 11:07:56 usb 1-1: configuration #1 chosen from 1 choice 2014-06-05 11:07:56 hub 1-1:1.0: USB hub found 2014-06-05 11:07:56 hub 1-1:1.0: 6 ports detected 2014-06-05 11:07:56 usb 2-1: new high speed USB device number 2 using ehci_hcd 2014-06-05 11:07:56 i8042.c: Can't read CTR while initializing i8042. 2014-06-05 11:07:56 i8042: probe of i8042 failed with error -5 2014-06-05 11:07:56 mice: PS/2 mouse device common for all mice 2014-06-05 11:07:56 rtc_cmos 00:06: RTC can wake from S4 2014-06-05 11:07:56 rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0 2014-06-05 11:07:56 rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs 2014-06-05 11:07:56 cpuidle: using governor ladder 2014-06-05 11:07:56 cpuidle: using governor menu 2014-06-05 11:07:56 EFI Variables Facility v0.08 2004-May-17 2014-06-05 11:07:56 usbcore: registered new interface driver hiddev 2014-06-05 11:07:56 usbcore: registered new interface driver usbhid 2014-06-05 11:07:56 usbhid: v2.6:USB HID core driver 2014-06-05 11:07:56 usb 2-1: New USB device found, idVendor=8087, idProduct=0024 2014-06-05 11:07:56 usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 2014-06-05 11:07:56 usb 2-1: configuration #1 chosen from 1 choice 2014-06-05 11:07:56 hub 2-1:1.0: USB hub found 2014-06-05 11:07:56 hub 2-1:1.0: 8 ports detected 2014-06-05 11:07:56 GRE over IPv4 demultiplexor driver 2014-06-05 11:07:56 TCP cubic registered 2014-06-05 11:07:56 Initializing XFRM netlink socket 2014-06-05 11:07:56 NET: Registered protocol family 17 2014-06-05 11:07:56 registered taskstats version 1 2014-06-05 11:07:56 rtc_cmos 00:06: setting system clock to 2014-06-05 18:07:56 UTC (1401991676) 2014-06-05 11:07:56 Initalizing network drop monitor service 2014-06-05 11:07:56 Freeing unused kernel memory: 1280k freed 2014-06-05 11:07:56 Write protecting the kernel read-only data: 10240k 2014-06-05 11:07:56 Freeing unused kernel memory: 788k freed 2014-06-05 11:07:56 Freeing unused kernel memory: 1584k freed 2014-06-05 11:07:56 udev: starting version 147 2014-06-05 11:07:56 mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) 2014-06-05 11:07:56 mlx4_core: Initializing 0000:01:00.0 2014-06-05 11:07:56 mlx4_core 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26 2014-06-05 11:07:56 usb 2-1.4: new full speed USB device number 3 using ehci_hcd 2014-06-05 11:07:56 usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10 2014-06-05 11:07:56 usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 2014-06-05 11:07:56 usb 2-1.4: Product: Virtual Keyboard and Mouse 2014-06-05 11:07:56 usb 2-1.4: Manufacturer: American Megatrends Inc. 2014-06-05 11:07:56 usb 2-1.4: SerialNumber: serial 2014-06-05 11:07:56 usb 2-1.4: configuration #1 chosen from 1 choice 2014-06-05 11:07:56 input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input2 2014-06-05 11:07:56 generic-usb 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0 2014-06-05 11:07:56 input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input3 2014-06-05 11:07:56 generic-usb 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1 2014-06-05 11:07:56 hub 2-1:1.0: over-current change on port 1 2014-06-05 11:07:58 mlx4_core 0000:01:00.0: 64B EQEs/CQEs supported by the device but not enabled 2014-06-05 11:07:59 mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011) 2014-06-05 11:07:59 pps_core: LinuxPPS API ver. 1 registered 2014-06-05 11:07:59 pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> 2014-06-05 11:07:59 PTP clock support registered 2014-06-05 11:07:59 dca service started, version 1.12.1 2014-06-05 11:07:59 igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k 2014-06-05 11:07:59 igb: Copyright (c) 2007-2013 Intel Corporation. 2014-06-05 11:07:59 igb 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 2014-06-05 11:07:59 igb 0000:06:00.0: added PHC on eth0 2014-06-05 11:07:59 igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection 2014-06-05 11:07:59 igb 0000:06:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:3c:e2:28 2014-06-05 11:07:59 igb 0000:06:00.0: eth0: PBA No: 100000-000 2014-06-05 11:07:59 igb 0000:06:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) 2014-06-05 11:07:59 igb 0000:06:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17 2014-06-05 11:07:59 igb 0000:06:00.1: added PHC on eth1 2014-06-05 11:07:59 igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection 2014-06-05 11:07:59 igb 0000:06:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:3c:e2:29 2014-06-05 11:07:59 igb 0000:06:00.1: eth1: PBA No: 100000-000 2014-06-05 11:07:59 igb 0000:06:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) 2014-06-05 11:08:03 igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX 2014-06-05 11:08:07 RPC: Registered named UNIX socket transport module. 2014-06-05 11:08:07 RPC: Registered udp transport module. 2014-06-05 11:08:07 RPC: Registered tcp transport module. 2014-06-05 11:08:07 RPC: Registered tcp NFSv4.1 backchannel transport module. 2014-06-05 11:08:07 Slow work thread pool: Starting up 2014-06-05 11:08:07 Slow work thread pool: Ready 2014-06-05 11:08:07 FS-Cache: Loaded 2014-06-05 11:08:07 NFS: Registering the id_resolver key type 2014-06-05 11:08:07 FS-Cache: Netfs 'nfs' registered for caching 2014-06-05 11:08:07 NET: Registered protocol family 10 2014-06-05 11:08:07 lo: Disabled Privacy Extensions 2014-06-05 11:08:07 dracut: Mounted root filesystem 192.168.64.201:/tftpboot/images/lscratche.x86_64 2014-06-05 11:08:07 SELinux: Disabled at runtime. 2014-06-05 11:08:07 type=1404 audit(1401991687.825:2): selinux=0 auid=4294967295 ses=4294967295 2014-06-05 11:08:07 dracut: 2014-06-05 11:08:08 dracut: Switching root 2014-06-05 11:08:08 rc.nfsroot-kdump: kernel was not booted via kexec 2014-06-05 11:08:08 rc.nfsroot-none: root file system is read-only 2014-06-05 11:08:08 rc.nfsroot-aufs: aufs is unavailable 2014-06-05 11:08:08 rc.nfsroot-unionfs: unionfs is unavailable 2014-06-05 11:08:08 rc.nfsroot-bind: mounting ramdisk with mode=755,nr_inodes=0 2014-06-05 11:08:08 rc.nfsroot-bind: copying /admin to ramdisk 2014-06-05 11:08:08 rc.nfsroot-bind: copying /cgroup to ramdisk 2014-06-05 11:08:08 rc.nfsroot-bind: copying /etc to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /mnt to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /p to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /root to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/account to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/cache/hald to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/cfengine to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/db/sudo to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lib/dbus to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lib/net-snmp to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lib/nfs to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lib/ntp to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lib/syslog-ng to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/lock to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/log to ramdisk 2014-06-05 11:08:09 rc.nfsroot-bind: copying /var/run to ramdisk 2014-06-05 11:08:10 rc.nfsroot-bind: copying /var/spool to ramdisk 2014-06-05 11:08:10 rc.nfsroot-bind: copying /var/cache/opensm to ramdisk 2014-06-05 11:08:10 rc.nfsroot-bind: copying /var/lib/heartbeat to ramdisk 2014-06-05 11:08:10 rc.nfsroot-bind: passing control to /etc/rc.nfsroot-init 2014-06-05 11:08:10 rc.nfsroot-init: setting up /etc/sysconfig/network 2014-06-05 11:08:10 rc.nfsroot-init: creating /etc/sysconfig/network 2014-06-05 11:08:10 rc.nfsroot-init: setting HOSTNAME=porter-mds1 2014-06-05 11:08:10 rc.nfsroot-init: creating /etc/sysconfig/network-scripts/ifcfg-eth0 2014-06-05 11:08:10 rc.nfsroot-init: executing /etc/rc.nfsroot.d/bootstrap 2014-06-05 11:08:10 Running: /etc/rc.nfsroot.d/bootstrap 2014-06-05 11:08:10 5 Jun 11:08:10 ntpdate[1075]: step time server 192.168.64.201 offset 0.043200 sec 2014-06-05 11:08:11 Attempting to mount /var/cfengine/inputs 2014-06-05 11:08:13 Running: copy of passwd and group files 2014-06-05 11:08:13 Running: cfagent 2014-06-05 11:08:13 PRECONF: Running cf.preconf 2014-06-05 11:08:16 cfengine:porter-mds1:ceservices: Turning off nfs 2014-06-05 11:08:16 cfengine:porter-mds1:ceservices: Turning off powerman 2014-06-05 11:08:16 cfengine:porter-mds1:ceservices: Turning off sendmail 2014-06-05 11:08:16 cfengine:porter-mds1:ceservices: Turning off sshd 2014-06-05 11:08:17 cfengine:porter-mds1:ceservices: Turning off lsi_mrdsnmpd 2014-06-05 11:08:17 cfengine:porter-mds1:ceservices: Turning on multipathd 2014-06-05 11:08:17 cfengine:porter-mds1:ceservices: Turning on zfs 2014-06-05 11:08:17 Running: /usr/sbin/make_interfaces -force 2014-06-05 11:08:17 Finished running /etc/rc.nfsroot.d/bootstrap 2014-06-05 11:08:17 rc.nfsroot-init: passing control to /sbin/init 2014-06-05 11:08:18 ^I^IWelcome to Red Hat Enterprise Linux Server 2014-06-05 11:08:18 Starting udev: udev: starting version 147 2014-06-05 11:08:18 mpt2sas version 15.101.00.00 loaded 2014-06-05 11:08:18 scsi0 : Fusion MPT SAS Host 2014-06-05 11:08:18 mpt2sas 0000:02:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32 2014-06-05 11:08:18 mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131982608 kB) 2014-06-05 11:08:18 mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 116 2014-06-05 11:08:18 mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 117 2014-06-05 11:08:18 mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 118 2014-06-05 11:08:18 mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 119 2014-06-05 11:08:18 mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 120 2014-06-05 11:08:18 mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 121 2014-06-05 11:08:18 mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 122 2014-06-05 11:08:18 mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 123 2014-06-05 11:08:18 mpt2sas0-msix8: PCI-MSI-X enabled: IRQ 124 2014-06-05 11:08:18 mpt2sas0-msix9: PCI-MSI-X enabled: IRQ 125 2014-06-05 11:08:18 mpt2sas0-msix10: PCI-MSI-X enabled: IRQ 126 2014-06-05 11:08:18 mpt2sas0-msix11: PCI-MSI-X enabled: IRQ 127 2014-06-05 11:08:18 mpt2sas0-msix12: PCI-MSI-X enabled: IRQ 128 2014-06-05 11:08:18 mpt2sas0-msix13: PCI-MSI-X enabled: IRQ 129 2014-06-05 11:08:18 mpt2sas0-msix14: PCI-MSI-X enabled: IRQ 130 2014-06-05 11:08:18 mpt2sas0-msix15: PCI-MSI-X enabled: IRQ 131 2014-06-05 11:08:18 mpt2sas0: iomem(0x00000000d0c40000), mapped(0xffffc9001d5a0000), size(65536) 2014-06-05 11:08:18 mpt2sas0: ioport(0x0000000000003000), size(256) 2014-06-05 11:08:18 mpt2sas0: sending message unit reset !! 2014-06-05 11:08:18 mpt2sas0: message unit reset: SUCCESS 2014-06-05 11:08:18 mpt2sas0: Allocated physical memory: size(18614 kB) 2014-06-05 11:08:18 mpt2sas0: Current Controller Queue Depth(9981), Max Controller Queue Depth(10240) 2014-06-05 11:08:18 mpt2sas0: Scatter Gather Elements per IO(256) 2014-06-05 11:08:18 mpt2sas0: LSISAS2308: FWVersion(09.00.02.00), ChipRevision(0x01), BiosVersion(07.17.03.00) 2014-06-05 11:08:18 mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) 2014-06-05 11:08:18 mpt2sas0: sending port enable !! 2014-06-05 11:08:18 scsi1 : Fusion MPT SAS Host 2014-06-05 11:08:18 mpt2sas 0000:04:00.0: PCI INT A -> GSI 42 (level, low) -> IRQ 42 2014-06-05 11:08:18 mpt2sas1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131982608 kB) 2014-06-05 11:08:18 mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 132 2014-06-05 11:08:18 mpt2sas1-msix1: PCI-MSI-X enabled: IRQ 133 2014-06-05 11:08:18 mpt2sas1-msix2: PCI-MSI-X enabled: IRQ 134 2014-06-05 11:08:18 mpt2sas1-msix3: PCI-MSI-X enabled: IRQ 135 2014-06-05 11:08:18 mpt2sas1-msix4: PCI-MSI-X enabled: IRQ 136 2014-06-05 11:08:18 mpt2sas1-msix5: PCI-MSI-X enabled: IRQ 137 2014-06-05 11:08:18 mpt2sas1-msix6: PCI-MSI-X enabled: IRQ 138 2014-06-05 11:08:18 mpt2sas1-msix7: PCI-MSI-X enabled: IRQ 139 2014-06-05 11:08:18 mpt2sas1-msix8: PCI-MSI-X enabled: IRQ 140 2014-06-05 11:08:18 mpt2sas1-msix9: PCI-MSI-X enabled: IRQ 141 2014-06-05 11:08:18 mpt2sas1-msix10: PCI-MSI-X enabled: IRQ 142 2014-06-05 11:08:18 mpt2sas1-msix11: PCI-MSI-X enabled: IRQ 143 2014-06-05 11:08:18 mpt2sas1-msix12: PCI-MSI-X enabled: IRQ 144 2014-06-05 11:08:18 mpt2sas1-msix13: PCI-MSI-X enabled: IRQ 145 2014-06-05 11:08:18 mpt2sas1-msix14: PCI-MSI-X enabled: IRQ 146 2014-06-05 11:08:18 mpt2sas1-msix15: PCI-MSI-X enabled: IRQ 147 2014-06-05 11:08:18 mpt2sas1: iomem(0x00000000d0a40000), mapped(0xffffc9001d320000), size(65536) 2014-06-05 11:08:18 mpt2sas1: ioport(0x0000000000002000), size(256) 2014-06-05 11:08:18 mpt2sas1: sending message unit reset !! 2014-06-05 11:08:18 mpt2sas1: message unit reset: SUCCESS 2014-06-05 11:08:18 mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b003e27230), phys(8) 2014-06-05 11:08:18 mpt2sas0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500093d000a1e03f), phys(37) 2014-06-05 11:08:18 scsi 0:0:0:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 mpt2sas1: Allocated physical memory: size(18614 kB) 2014-06-05 11:08:18 mpt2sas1: Current Controller Queue Depth(9981), Max Controller Queue Depth(10240) 2014-06-05 11:08:18 mpt2sas1: Scatter Gather Elements per IO(256) 2014-06-05 11:08:18 scsi 0:0:0:0: SSP: handle(0x000a), sas_addr(0x5000c5004b8982b9), phy(0), device_name(0x00c50050b882894b) 2014-06-05 11:08:18 scsi 0:0:0:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(0) 2014-06-05 11:08:18 scsi 0:0:0:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 scsi 0:0:1:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 scsi 0:0:1:0: SSP: handle(0x000b), sas_addr(0x5000c5004b8ac4d5), phy(1), device_name(0x00c50050d4c48a4b) 2014-06-05 11:08:18 scsi 0:0:1:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(1) 2014-06-05 11:08:18 scsi 0:0:1:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 scsi 0:0:2:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 scsi 0:0:2:0: SSP: handle(0x000c), sas_addr(0x5000c500486a44fd), phy(2), device_name(0x00c50050fc446a48) 2014-06-05 11:08:18 scsi 0:0:2:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(2) 2014-06-05 11:08:18 scsi 0:0:2:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 scsi 0:0:3:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 scsi 0:0:3:0: SSP: handle(0x000d), sas_addr(0x5000c5004b898575), phy(3), device_name(0x00c500507485894b) 2014-06-05 11:08:18 scsi 0:0:3:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(3) 2014-06-05 11:08:18 scsi 0:0:3:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 mpt2sas1: LSISAS2308: FWVersion(09.00.02.00), ChipRevision(0x01), BiosVersion(07.17.03.00) 2014-06-05 11:08:18 mpt2sas1: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) 2014-06-05 11:08:18 mpt2sas1: sending port enable !! 2014-06-05 11:08:18 scsi 0:0:4:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 scsi 0:0:4:0: SSP: handle(0x000e), sas_addr(0x5000c5004b996b99), phy(4), device_name(0x00c50050986b994b) 2014-06-05 11:08:18 scsi 0:0:4:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(4) 2014-06-05 11:08:18 scsi 0:0:4:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 scsi 0:0:5:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:18 scsi 0:0:5:0: SSP: handle(0x000f), sas_addr(0x5000c500486a4645), phy(5), device_name(0x00c5005044466a48) 2014-06-05 11:08:18 scsi 0:0:5:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(5) 2014-06-05 11:08:18 scsi 0:0:5:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:18 scsi 0:0:6:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:6:0: SSP: handle(0x0010), sas_addr(0x5000c5004b9743a5), phy(6), device_name(0x00c50050a443974b) 2014-06-05 11:08:19 scsi 0:0:6:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(6) 2014-06-05 11:08:19 scsi 0:0:6:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:7:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:7:0: SSP: handle(0x0011), sas_addr(0x5000c500486a70ed), phy(7), device_name(0x00c50050ec706a48) 2014-06-05 11:08:19 scsi 0:0:7:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(7) 2014-06-05 11:08:19 scsi 0:0:7:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:8:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:8:0: SSP: handle(0x0012), sas_addr(0x5000c5004b9404e5), phy(8), device_name(0x00c50050e404944b) 2014-06-05 11:08:19 scsi 0:0:8:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(8) 2014-06-05 11:08:19 scsi 0:0:8:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:9:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:9:0: SSP: handle(0x0013), sas_addr(0x5000c500486ba2b9), phy(9), device_name(0x00c50050b8a26b48) 2014-06-05 11:08:19 scsi 0:0:9:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(9) 2014-06-05 11:08:19 scsi 0:0:9:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:10:0: Direct-Access SMARTMOD SG9XCA2E200GE01 F03E PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:10:0: SSP: handle(0x0014), sas_addr(0x500051610010080d), phy(23), device_name(0x615100500c081000) 2014-06-05 11:08:19 scsi 0:0:10:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(23) 2014-06-05 11:08:19 scsi 0:0:10:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:11:0: Enclosure NEWISYS NDS2241 0418 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:11:0: SSP: handle(0x0016), sas_addr(0x500093d000a1e03e), phy(36), device_name(0x0000000000000000) 2014-06-05 11:08:19 scsi 0:0:11:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(24) 2014-06-05 11:08:19 scsi 0:0:11:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 mpt2sas0: expander_add: handle(0x0017), parent(0x0002), sas_addr(0x500093d000aa703f), phys(37) 2014-06-05 11:08:19 scsi 0:0:12:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:12:0: SSP: handle(0x0018), sas_addr(0x5000c5004b9411d1), phy(0), device_name(0x00c50050d011944b) 2014-06-05 11:08:19 scsi 0:0:12:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(0) 2014-06-05 11:08:19 scsi 0:0:12:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:13:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:13:0: SSP: handle(0x0019), sas_addr(0x5000c5004b89871d), phy(1), device_name(0x00c500501c87894b) 2014-06-05 11:08:19 scsi 0:0:13:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(1) 2014-06-05 11:08:19 scsi 0:0:13:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:14:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:14:0: SSP: handle(0x001a), sas_addr(0x5000c500486a5ce5), phy(2), device_name(0x00c50050e45c6a48) 2014-06-05 11:08:19 scsi 0:0:14:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(2) 2014-06-05 11:08:19 scsi 0:0:14:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:15:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:15:0: SSP: handle(0x001b), sas_addr(0x5000c500486bdca5), phy(3), device_name(0x00c50050a4dc6b48) 2014-06-05 11:08:19 scsi 0:0:15:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(3) 2014-06-05 11:08:19 scsi 0:0:15:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:16:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:16:0: SSP: handle(0x001c), sas_addr(0x5000c5004b94337d), phy(4), device_name(0x00c500507c33944b) 2014-06-05 11:08:19 scsi 0:0:16:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(4) 2014-06-05 11:08:19 scsi 0:0:16:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:17:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:17:0: SSP: handle(0x001d), sas_addr(0x5000c500486afd01), phy(5), device_name(0x00c5005000fd6a48) 2014-06-05 11:08:19 scsi 0:0:17:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(5) 2014-06-05 11:08:19 scsi 0:0:17:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:18:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:18:0: SSP: handle(0x001e), sas_addr(0x5000c5004b940785), phy(6), device_name(0x00c500508407944b) 2014-06-05 11:08:19 scsi 0:0:18:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(6) 2014-06-05 11:08:19 scsi 0:0:18:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:19:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:19:0: SSP: handle(0x001f), sas_addr(0x5000c500486af15d), phy(7), device_name(0x00c500505cf16a48) 2014-06-05 11:08:19 scsi 0:0:19:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(7) 2014-06-05 11:08:19 scsi 0:0:19:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:20:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:20:0: SSP: handle(0x0020), sas_addr(0x5000c500487588d9), phy(8), device_name(0x00c50050d8887548) 2014-06-05 11:08:19 scsi 0:0:20:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(8) 2014-06-05 11:08:19 scsi 0:0:20:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:21:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:21:0: SSP: handle(0x0021), sas_addr(0x5000c50048699bf1), phy(9), device_name(0x00c50050f09b6948) 2014-06-05 11:08:19 scsi 0:0:21:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(9) 2014-06-05 11:08:19 scsi 0:0:21:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 0:0:22:0: Enclosure NEWISYS NDS2241 0418 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 0:0:22:0: SSP: handle(0x0023), sas_addr(0x500093d000aa703e), phy(36), device_name(0x0000000000000000) 2014-06-05 11:08:19 scsi 0:0:22:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(24) 2014-06-05 11:08:19 scsi 0:0:22:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 mpt2sas0: port enable: SUCCESS 2014-06-05 11:08:19 mpt2sas1: host_add: handle(0x0001), sas_addr(0x500605b0047a1160), phys(8) 2014-06-05 11:08:19 mpt2sas1: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500093d000a1e23f), phys(37) 2014-06-05 11:08:19 mpt2sas1: expander_add: handle(0x0017), parent(0x0002), sas_addr(0x500093d000aa723f), phys(37) 2014-06-05 11:08:19 mpt2sas1: port enable: SUCCESS 2014-06-05 11:08:19 scsi 1:0:0:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:0:0: SSP: handle(0x0018), sas_addr(0x5000c5004b9411d2), phy(0), device_name(0x00c50050d011944b) 2014-06-05 11:08:19 scsi 1:0:0:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(0) 2014-06-05 11:08:19 scsi 1:0:0:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:1:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:1:0: SSP: handle(0x000a), sas_addr(0x5000c5004b8982ba), phy(0), device_name(0x00c50050b882894b) 2014-06-05 11:08:19 scsi 1:0:1:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(0) 2014-06-05 11:08:19 scsi 1:0:1:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:2:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:2:0: SSP: handle(0x000b), sas_addr(0x5000c5004b8ac4d6), phy(1), device_name(0x00c50050d4c48a4b) 2014-06-05 11:08:19 scsi 1:0:2:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(1) 2014-06-05 11:08:19 scsi 1:0:2:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:3:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:3:0: SSP: handle(0x000c), sas_addr(0x5000c500486a44fe), phy(2), device_name(0x00c50050fc446a48) 2014-06-05 11:08:19 scsi 1:0:3:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(2) 2014-06-05 11:08:19 scsi 1:0:3:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:4:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:4:0: SSP: handle(0x000d), sas_addr(0x5000c5004b898576), phy(3), device_name(0x00c500507485894b) 2014-06-05 11:08:19 scsi 1:0:4:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(3) 2014-06-05 11:08:19 scsi 1:0:4:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:5:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:5:0: SSP: handle(0x000e), sas_addr(0x5000c5004b996b9a), phy(4), device_name(0x00c50050986b994b) 2014-06-05 11:08:19 scsi 1:0:5:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(4) 2014-06-05 11:08:19 scsi 1:0:5:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:6:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:6:0: SSP: handle(0x000f), sas_addr(0x5000c500486a4646), phy(5), device_name(0x00c5005044466a48) 2014-06-05 11:08:19 scsi 1:0:6:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(5) 2014-06-05 11:08:19 scsi 1:0:6:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:7:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:7:0: SSP: handle(0x0010), sas_addr(0x5000c5004b9743a6), phy(6), device_name(0x00c50050a443974b) 2014-06-05 11:08:19 scsi 1:0:7:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(6) 2014-06-05 11:08:19 scsi 1:0:7:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:8:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:8:0: SSP: handle(0x0011), sas_addr(0x5000c500486a70ee), phy(7), device_name(0x00c50050ec706a48) 2014-06-05 11:08:19 scsi 1:0:8:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(7) 2014-06-05 11:08:19 scsi 1:0:8:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:9:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:19 scsi 1:0:9:0: SSP: handle(0x0012), sas_addr(0x5000c5004b9404e6), phy(8), device_name(0x00c50050e404944b) 2014-06-05 11:08:19 scsi 1:0:9:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(8) 2014-06-05 11:08:19 scsi 1:0:9:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:19 scsi 1:0:10:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:10:0: SSP: handle(0x0013), sas_addr(0x5000c500486ba2ba), phy(9), device_name(0x00c50050b8a26b48) 2014-06-05 11:08:20 scsi 1:0:10:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(9) 2014-06-05 11:08:20 scsi 1:0:10:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:11:0: Direct-Access SMARTMOD SG9XCA2E200GE01 F03E PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:11:0: SSP: handle(0x0014), sas_addr(0x500051610010080e), phy(23), device_name(0x615100500c081000) 2014-06-05 11:08:20 scsi 1:0:11:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(23) 2014-06-05 11:08:20 scsi 1:0:11:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:12:0: Enclosure NEWISYS NDS2241 0418 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:12:0: SSP: handle(0x0016), sas_addr(0x500093d000a1e23e), phy(36), device_name(0x0000000000000000) 2014-06-05 11:08:20 scsi 1:0:12:0: SSP: enclosure_logical_id(0x500093d000a1e000), slot(24) 2014-06-05 11:08:20 scsi 1:0:12:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:13:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:13:0: SSP: handle(0x0019), sas_addr(0x5000c5004b89871e), phy(1), device_name(0x00c500501c87894b) 2014-06-05 11:08:20 scsi 1:0:13:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(1) 2014-06-05 11:08:20 scsi 1:0:13:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:14:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:14:0: SSP: handle(0x001a), sas_addr(0x5000c500486a5ce6), phy(2), device_name(0x00c50050e45c6a48) 2014-06-05 11:08:20 scsi 1:0:14:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(2) 2014-06-05 11:08:20 scsi 1:0:14:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:15:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:15:0: SSP: handle(0x001b), sas_addr(0x5000c500486bdca6), phy(3), device_name(0x00c50050a4dc6b48) 2014-06-05 11:08:20 scsi 1:0:15:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(3) 2014-06-05 11:08:20 scsi 1:0:15:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:16:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:16:0: SSP: handle(0x001c), sas_addr(0x5000c5004b94337e), phy(4), device_name(0x00c500507c33944b) 2014-06-05 11:08:20 scsi 1:0:16:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(4) 2014-06-05 11:08:20 scsi 1:0:16:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:17:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:17:0: SSP: handle(0x001d), sas_addr(0x5000c500486afd02), phy(5), device_name(0x00c5005000fd6a48) 2014-06-05 11:08:20 scsi 1:0:17:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(5) 2014-06-05 11:08:20 scsi 1:0:17:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:18:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:18:0: SSP: handle(0x001e), sas_addr(0x5000c5004b940786), phy(6), device_name(0x00c500508407944b) 2014-06-05 11:08:20 scsi 1:0:18:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(6) 2014-06-05 11:08:20 scsi 1:0:18:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:19:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:19:0: SSP: handle(0x001f), sas_addr(0x5000c500486af15e), phy(7), device_name(0x00c500505cf16a48) 2014-06-05 11:08:20 scsi 1:0:19:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(7) 2014-06-05 11:08:20 scsi 1:0:19:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:20:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:20:0: SSP: handle(0x0020), sas_addr(0x5000c500487588da), phy(8), device_name(0x00c50050d8887548) 2014-06-05 11:08:20 scsi 1:0:20:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(8) 2014-06-05 11:08:20 scsi 1:0:20:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:21:0: Direct-Access SEAGATE ST9300603SS 0006 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:21:0: SSP: handle(0x0021), sas_addr(0x5000c50048699bf2), phy(9), device_name(0x00c50050f09b6948) 2014-06-05 11:08:20 scsi 1:0:21:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(9) 2014-06-05 11:08:20 scsi 1:0:21:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 scsi 1:0:22:0: Enclosure NEWISYS NDS2241 0418 PQ: 0 ANSI: 5 2014-06-05 11:08:20 scsi 1:0:22:0: SSP: handle(0x0023), sas_addr(0x500093d000aa723e), phy(36), device_name(0x0000000000000000) 2014-06-05 11:08:20 scsi 1:0:22:0: SSP: enclosure_logical_id(0x500093d000aa7000), slot(24) 2014-06-05 11:08:20 scsi 1:0:22:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) 2014-06-05 11:08:20 ioatdma: Intel(R) QuickData Technology Driver 4.00 2014-06-05 11:08:20 ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31 2014-06-05 11:08:20 igb 0000:06:00.0: DCA enabled 2014-06-05 11:08:20 igb 0000:06:00.1: DCA enabled 2014-06-05 11:08:20 ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39 2014-06-05 11:08:20 ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31 2014-06-05 11:08:20 ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39 2014-06-05 11:08:20 ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31 2014-06-05 11:08:20 ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39 2014-06-05 11:08:20 ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31 2014-06-05 11:08:20 ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39 2014-06-05 11:08:20 ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55 2014-06-05 11:08:20 ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63 2014-06-05 11:08:20 ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55 2014-06-05 11:08:20 ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63 2014-06-05 11:08:20 ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55 2014-06-05 11:08:20 ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63 2014-06-05 11:08:20 ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55 2014-06-05 11:08:20 ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63 2014-06-05 11:08:20 ACPI: resource (null) [io 0x0428-0x042f] conflicts with ACPI region PMIO [io 0x400-0x47f] 2014-06-05 11:08:20 ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver 2014-06-05 11:08:20 ACPI: resource (null) [io 0x0500-0x053f] conflicts with ACPI region GPIO [io 0x500-0x563] 2014-06-05 11:08:20 ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver 2014-06-05 11:08:20 lpc_ich: Resource conflict(s) found affecting gpio_ich 2014-06-05 11:08:20 ahci 0000:00:1f.2: PCI INT B -> GSI 21 (level, low) -> IRQ 21 2014-06-05 11:08:20 ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode 2014-06-05 11:08:20 ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 2014-06-05 11:08:20 scsi2 : ahci 2014-06-05 11:08:20 scsi3 : ahci 2014-06-05 11:08:20 scsi4 : ahci 2014-06-05 11:08:20 scsi5 : ahci 2014-06-05 11:08:20 scsi6 : ahci 2014-06-05 11:08:20 scsi7 : ahci 2014-06-05 11:08:20 ata1: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00100 irq 164 2014-06-05 11:08:20 ata2: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00180 irq 164 2014-06-05 11:08:20 ata3: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00200 irq 164 2014-06-05 11:08:20 ata4: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00280 irq 164 2014-06-05 11:08:20 ata5: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00300 irq 164 2014-06-05 11:08:20 ata6: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00380 irq 164 2014-06-05 11:08:21 ata2: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 ata4: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 ata3: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 ata5: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 ata1: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 ata6: SATA link down (SStatus 0 SControl 300) 2014-06-05 11:08:21 i801_smbus 0000:00:1f.3: PCI INT C -> GSI 23 (level, low) -> IRQ 23 2014-06-05 11:08:21 ACPI: resource 0000:00:1f.3 [io 0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f] 2014-06-05 11:08:21 ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver 2014-06-05 11:08:21 EDAC MC: Ver: 2.1.0 May 16 2014 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 2014-06-05 11:08:21 EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 2014-06-05 11:08:21 EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0 2014-06-05 11:08:21 EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0 2014-06-05 11:08:21 EDAC sbridge: Driver loaded. 2014-06-05 11:08:21 ACPI: WMI: Mapper loaded 2014-06-05 11:08:22 sd 0:0:0:0: [sda] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:1:0: [sdb] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:2:0: [sdc] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:3:0: [sdd] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:4:0: [sde] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:7:0: [sdh] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:8:0: [sdi] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:9:0: [sdj] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:5:0: [sdf] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:12:0: [sdl] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:13:0: [sdm] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:6:0: [sdg] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:14:0: [sdn] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:15:0: [sdo] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:16:0: [sdp] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:10:0: [sdk] 390721968 512-byte logical blocks: (200 GB/186 GiB) 2014-06-05 11:08:22 sd 0:0:17:0: [sdq] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:18:0: [sdr] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:19:0: [sds] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:20:0: [sdt] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:2:0: [sdc] Write Protect is off 2014-06-05 11:08:22 sd 0:0:1:0: [sdb] Write Protect is off 2014-06-05 11:08:22 sd 0:0:21:0: [sdu] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:3:0: [sdd] Write Protect is off 2014-06-05 11:08:22 sd 0:0:4:0: [sde] Write Protect is off 2014-06-05 11:08:22 sd 1:0:1:0: [sdw] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:7:0: [sdh] Write Protect is off 2014-06-05 11:08:22 sd 0:0:10:0: [sdk] Write Protect is off 2014-06-05 11:08:22 sd 0:0:8:0: [sdi] Write Protect is off 2014-06-05 11:08:22 sd 0:0:9:0: [sdj] Write Protect is off 2014-06-05 11:08:22 sd 0:0:5:0: [sdf] Write Protect is off 2014-06-05 11:08:22 sd 0:0:10:0: [sdk] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA 2014-06-05 11:08:22 sd 0:0:13:0: [sdm] Write Protect is off 2014-06-05 11:08:22 sd 0:0:6:0: [sdg] Write Protect is off 2014-06-05 11:08:22 sd 0:0:14:0: [sdn] Write Protect is off 2014-06-05 11:08:22 sd 0:0:12:0: [sdl] Write Protect is off 2014-06-05 11:08:22 sd 0:0:15:0: [sdo] Write Protect is off 2014-06-05 11:08:22 sd 0:0:16:0: [sdp] Write Protect is off 2014-06-05 11:08:22 sd 1:0:11:0: [sdag] 390721968 512-byte logical blocks: (200 GB/186 GiB) 2014-06-05 11:08:22 sd 1:0:0:0: [sdv] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:4:0: [sdz] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:3:0: [sdy] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:2:0: [sdx] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:17:0: [sdq] Write Protect is off 2014-06-05 11:08:22 sd 0:0:18:0: [sdr] Write Protect is off 2014-06-05 11:08:22 sd 0:0:19:0: [sds] Write Protect is off 2014-06-05 11:08:22 sd 0:0:20:0: [sdt] Write Protect is off 2014-06-05 11:08:22 sd 0:0:21:0: [sdu] Write Protect is off 2014-06-05 11:08:22 sd 1:0:6:0: [sdab] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:1:0: [sdw] Write Protect is off 2014-06-05 11:08:22 sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:7:0: [sdac] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:5:0: [sdaa] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:8:0: [sdad] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:9:0: [sdae] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:13:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:10:0: [sdaf] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:14:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:15:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:16:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:11:0: [sdag] Write Protect is off 2014-06-05 11:08:22 sd 1:0:13:0: [sdah] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sdk: 2014-06-05 11:08:22 sd 1:0:15:0: [sdaj] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:14:0: [sdai] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:17:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:12:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:18:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:16:0: [sdak] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 0:0:21:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:19:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:11:0: [sdag] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA 2014-06-05 11:08:22 sd 0:0:20:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:17:0: [sdal] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:18:0: [sdam] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 sd 1:0:1:0: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:19:0: [sdan] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:20:0: [sdao] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:21:0: [sdap] 585937500 512-byte logical blocks: (300 GB/279 GiB) 2014-06-05 11:08:22 sd 1:0:8:0: [sdad] Write Protect is off 2014-06-05 11:08:22 sd 1:0:7:0: [sdac] Write Protect is off 2014-06-05 11:08:22 sd 1:0:3:0: [sdy] Write Protect is off 2014-06-05 11:08:22 sd 1:0:8:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:16:0: [sdak] Write Protect is off 2014-06-05 11:08:22 sd 1:0:18:0: [sdam] Write Protect is off 2014-06-05 11:08:22 sd 1:0:14:0: [sdai] Write Protect is off 2014-06-05 11:08:22 sd 1:0:17:0: [sdal] Write Protect is off 2014-06-05 11:08:22 sd 1:0:3:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:20:0: [sdao] Write Protect is off 2014-06-05 11:08:22 sd 1:0:20:0: [sdao] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:18:0: [sdam] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:19:0: [sdan] Write Protect is off 2014-06-05 11:08:22 sd 1:0:19:0: [sdan] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:10:0: [sdaf] Write Protect is off 2014-06-05 11:08:22 sd 1:0:2:0: [sdx] Write Protect is off 2014-06-05 11:08:22 sd 1:0:15:0: [sdaj] Write Protect is off 2014-06-05 11:08:22 sd 1:0:10:0: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:6:0: [sdab] Write Protect is off 2014-06-05 11:08:22 sd 1:0:5:0: [sdaa] Write Protect is off 2014-06-05 11:08:22 sd 1:0:6:0: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:17:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:14:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:16:0: [sdak] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:21:0: [sdap] Write Protect is off 2014-06-05 11:08:22 sd 1:0:21:0: [sdap] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:7:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:5:0: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:4:0: [sdz] Write Protect is off 2014-06-05 11:08:22 sd 1:0:2:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:0:0: [sdv] Write Protect is off 2014-06-05 11:08:22 sd 1:0:13:0: [sdah] Write Protect is off 2014-06-05 11:08:22 sd 1:0:9:0: [sdae] Write Protect is off 2014-06-05 11:08:22 sd 1:0:15:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:0:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:4:0: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 0:0:10:0: [sdk] Attached SCSI disk 2014-06-05 11:08:22 sdag: 2014-06-05 11:08:22 sd 0:0:0:0: [sda] Write Protect is off 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:9:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sd 1:0:13:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA 2014-06-05 11:08:22 sde: 2014-06-05 11:08:22 sdn: 2014-06-05 11:08:22 sdc: 2014-06-05 11:08:22 sdb: 2014-06-05 11:08:22 sdo: 2014-06-05 11:08:22 sdu: 2014-06-05 11:08:22 sdq: 2014-06-05 11:08:22 sdw: unknown partition table 2014-06-05 11:08:22 sdao: 2014-06-05 11:08:22 sdad: 2014-06-05 11:08:22 sdr: 2014-06-05 11:08:22 sdam: unknown partition table 2014-06-05 11:08:22 sdan: 2014-06-05 11:08:22 sdab: unknown partition table 2014-06-05 11:08:22 sd 0:0:15:0: [sdo] Attached SCSI disk 2014-06-05 11:08:22 sdal: 2014-06-05 11:08:22 sdi: unknown partition table 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 sd 1:0:1:0: [sdw] Attached SCSI disk 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 sd 1:0:11:0: [sdag] Attached SCSI disk 2014-06-05 11:08:22 sd 0:0:1:0: [sdb] Attached SCSI disk 2014-06-05 11:08:22 sdae: 2014-06-05 11:08:22 sdd: unknown partition table 2014-06-05 11:08:22 unknown partition table 2014-06-05 11:08:22 sd 0:0:14:0: [sdn] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:19:0: [sdan] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:21:0: [sdu] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:20:0: [sdao] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:6:0: [sdab] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sdv: unknown partition table 2014-06-05 11:08:23 sd 0:0:2:0: [sdc] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:17:0: [sdq] Attached SCSI disk 2014-06-05 11:08:23 sdai: 2014-06-05 11:08:23 sdaf: unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 1:0:8:0: [sdad] Attached SCSI disk 2014-06-05 11:08:23 sdah: unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sdg: unknown partition table 2014-06-05 11:08:23 sd 1:0:18:0: [sdam] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:3:0: [sdd] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:0:0: [sdv] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:17:0: [sdal] Attached SCSI disk 2014-06-05 11:08:23 sdt: unknown partition table 2014-06-05 11:08:23 sd 0:0:18:0: [sdr] Attached SCSI disk 2014-06-05 11:08:23 sdak: unknown partition table 2014-06-05 11:08:23 sdj: unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 0:0:8:0: [sdi] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 1:0:9:0: [sdae] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:6:0: [sdg] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:13:0: [sdah] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:10:0: [sdaf] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:14:0: [sdai] Attached SCSI disk 2014-06-05 11:08:23 sdp: 2014-06-05 11:08:23 sdx: 2014-06-05 11:08:23 sdl: 2014-06-05 11:08:23 sdm: unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sds: 2014-06-05 11:08:23 sdap: 2014-06-05 11:08:23 sdf: unknown partition table 2014-06-05 11:08:23 sdh: 2014-06-05 11:08:23 sd 0:0:9:0: [sdj] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:16:0: [sdak] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:20:0: [sdt] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sdac: 2014-06-05 11:08:23 sd 0:0:5:0: [sdf] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:4:0: [sde] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:7:0: [sdh] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sdz: 2014-06-05 11:08:23 sd 0:0:19:0: [sds] Attached SCSI disk 2014-06-05 11:08:23 sdaj: unknown partition table 2014-06-05 11:08:23 sd 1:0:21:0: [sdap] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 0:0:13:0: [sdm] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:16:0: [sdp] Attached SCSI disk 2014-06-05 11:08:23 sdy: unknown partition table 2014-06-05 11:08:23 sd 0:0:12:0: [sdl] Attached SCSI disk 2014-06-05 11:08:23 sda: unknown partition table 2014-06-05 11:08:23 sd 1:0:2:0: [sdx] Attached SCSI disk 2014-06-05 11:08:23 sdaa: unknown partition table 2014-06-05 11:08:23 sd 1:0:3:0: [sdy] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 1:0:7:0: [sdac] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:15:0: [sdaj] Attached SCSI disk 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 unknown partition table 2014-06-05 11:08:23 sd 1:0:4:0: [sdz] Attached SCSI disk 2014-06-05 11:08:23 sd 1:0:5:0: [sdaa] Attached SCSI disk 2014-06-05 11:08:23 sd 0:0:0:0: [sda] Attached SCSI disk 2014-06-05 11:09:13 ses 0:0:11:0: Attached Enclosure device 2014-06-05 11:09:13 ses 0:0:22:0: Attached Enclosure device 2014-06-05 11:09:13 ses 1:0:12:0: Attached Enclosure device 2014-06-05 11:09:13 ses 1:0:22:0: Attached Enclosure device 2014-06-05 11:09:13 sd 0:0:0:0: Attached scsi generic sg0 type 0 2014-06-05 11:09:13 sd 0:0:1:0: Attached scsi generic sg1 type 0 2014-06-05 11:09:13 sd 0:0:2:0: Attached scsi generic sg2 type 0 2014-06-05 11:09:13 sd 0:0:3:0: Attached scsi generic sg3 type 0 2014-06-05 11:09:13 sd 0:0:4:0: Attached scsi generic sg4 type 0 2014-06-05 11:09:13 sd 0:0:5:0: Attached scsi generic sg5 type 0 2014-06-05 11:09:13 sd 0:0:6:0: Attached scsi generic sg6 type 0 2014-06-05 11:09:13 sd 0:0:7:0: Attached scsi generic sg7 type 0 2014-06-05 11:09:13 sd 0:0:8:0: Attached scsi generic sg8 type 0 2014-06-05 11:09:13 sd 0:0:9:0: Attached scsi generic sg9 type 0 2014-06-05 11:09:13 sd 0:0:10:0: Attached scsi generic sg10 type 0 2014-06-05 11:09:13 ses 0:0:11:0: Attached scsi generic sg11 type 13 2014-06-05 11:09:13 sd 0:0:12:0: Attached scsi generic sg12 type 0 2014-06-05 11:09:13 sd 0:0:13:0: Attached scsi generic sg13 type 0 2014-06-05 11:09:13 sd 0:0:14:0: Attached scsi generic sg14 type 0 2014-06-05 11:09:13 sd 0:0:15:0: Attached scsi generic sg15 type 0 2014-06-05 11:09:13 sd 0:0:16:0: Attached scsi generic sg16 type 0 2014-06-05 11:09:13 sd 0:0:17:0: Attached scsi generic sg17 type 0 2014-06-05 11:09:13 sd 0:0:18:0: Attached scsi generic sg18 type 0 2014-06-05 11:09:13 sd 0:0:19:0: Attached scsi generic sg19 type 0 2014-06-05 11:09:13 sd 0:0:20:0: Attached scsi generic sg20 type 0 2014-06-05 11:09:13 sd 0:0:21:0: Attached scsi generic sg21 type 0 2014-06-05 11:09:13 ses 0:0:22:0: Attached scsi generic sg22 type 13 2014-06-05 11:09:13 sd 1:0:0:0: Attached scsi generic sg23 type 0 2014-06-05 11:09:13 sd 1:0:1:0: Attached scsi generic sg24 type 0 2014-06-05 11:09:13 sd 1:0:2:0: Attached scsi generic sg25 type 0 2014-06-05 11:09:13 sd 1:0:3:0: Attached scsi generic sg26 type 0 2014-06-05 11:09:13 sd 1:0:4:0: Attached scsi generic sg27 type 0 2014-06-05 11:09:13 sd 1:0:5:0: Attached scsi generic sg28 type 0 2014-06-05 11:09:13 sd 1:0:6:0: Attached scsi generic sg29 type 0 2014-06-05 11:09:13 sd 1:0:7:0: Attached scsi generic sg30 type 0 2014-06-05 11:09:13 sd 1:0:8:0: Attached scsi generic sg31 type 0 2014-06-05 11:09:13 sd 1:0:9:0: Attached scsi generic sg32 type 0 2014-06-05 11:09:13 sd 1:0:10:0: Attached scsi generic sg33 type 0 2014-06-05 11:09:13 sd 1:0:11:0: Attached scsi generic sg34 type 0 2014-06-05 11:09:13 ses 1:0:12:0: Attached scsi generic sg35 type 13 2014-06-05 11:09:13 sd 1:0:13:0: Attached scsi generic sg36 type 0 2014-06-05 11:09:13 sd 1:0:14:0: Attached scsi generic sg37 type 0 2014-06-05 11:09:13 sd 1:0:15:0: Attached scsi generic sg38 type 0 2014-06-05 11:09:13 sd 1:0:16:0: Attached scsi generic sg39 type 0 2014-06-05 11:09:13 sd 1:0:17:0: Attached scsi generic sg40 type 0 2014-06-05 11:09:13 sd 1:0:18:0: Attached scsi generic sg41 type 0 2014-06-05 11:09:14 sd 1:0:19:0: Attached scsi generic sg42 type 0 2014-06-05 11:09:14 sd 1:0:20:0: Attached scsi generic sg43 type 0 2014-06-05 11:09:14 sd 1:0:21:0: Attached scsi generic sg44 type 0 2014-06-05 11:09:14 ses 1:0:22:0: Attached scsi generic sg45 type 13 2014-06-05 11:09:14 iTCO_vendor_support: vendor-support=0 2014-06-05 11:09:14 iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh 2014-06-05 11:09:14 iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS 2014-06-05 11:09:14 SPL: Loaded module v0.6.2-40.1 2014-06-05 11:09:14 zunicode: module license 'CDDL' taints kernel. 2014-06-05 11:09:14 Disabling lock debugging due to kernel taint 2014-06-05 11:09:14 ZFS: Loaded module v0.6.2-276.1, ZFS pool version 5000, ZFS filesystem version 5 2014-06-05 11:09:14 [ OK ] 2014-06-05 11:09:14 kvm: disabled by bios 2014-06-05 11:09:14 tun: Universal TUN/TAP device driver, 1.6 2014-06-05 11:09:14 tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> 2014-06-05 11:09:14 Setting hostname porter-mds1: [ OK ] 2014-06-05 11:09:14 device-mapper: uevent: version 1.0.3 2014-06-05 11:09:14 device-mapper: ioctl: 4.24.6-ioctl (2013-01-15) initialised: dm-devel@redhat.com 2014-06-05 11:09:14 device-mapper: multipath: version 1.5.0 loaded 2014-06-05 11:09:15 device-mapper: multipath round-robin: version 1.0.0 loaded 2014-06-05 11:09:16 Setting up Logical Volume Management: No volume groups found 2014-06-05 11:09:16 [ OK ] 2014-06-05 11:09:16 Checking filesystems 2014-06-05 11:09:16 Checking all file systems. 2014-06-05 11:09:16 [ OK ] 2014-06-05 11:09:16 Mounting local filesystems: [ OK ] 2014-06-05 11:09:16 touch: cannot touch `/.autorelabel': Read-only file system 2014-06-05 11:09:16 chmod: changing permissions of `/var/lib/random-seed': Read-only file system 2014-06-05 11:09:16 Enabling /etc/fstab swaps: [ OK ] 2014-06-05 11:09:16 init: rcS main process (1554) terminated with status 1 2014-06-05 11:09:16 Entering non-interactive startup 2014-06-05 11:09:16 Calling the system activity data collector (sadc)... 2014-06-05 11:09:17 Mounting automounted ZFS filesystems: [ OK ] 2014-06-05 11:09:17 Exporting ZFS filesystems: [ OK ] 2014-06-05 11:09:17 Starting cgconfig service: [ OK ] 2014-06-05 11:09:17 Loading OpenIB kernel modules:<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008) 2014-06-05 11:09:18 [ OK ] 2014-06-05 11:09:18 Starting multipathd daemon: [ OK ] 2014-06-05 11:09:18 Setting sysfs variables...done. 2014-06-05 11:09:18 Bringing up loopback interface: [ OK ] 2014-06-05 11:09:18 Bringing up interface ib0: ib0: enabling connected mode will cause multicast packet drops 2014-06-05 11:09:18 ib0: mtu > 4092 will cause multicast packet drops. 2014-06-05 11:09:18 ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 2014-06-05 11:09:18 ADDRCONF(NETDEV_UP): ib0: link is not ready 2014-06-05 11:09:20 ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 2014-06-05 11:09:20 [ OK ] 2014-06-05 11:09:21 Starting portreserve: [ OK ] 2014-06-05 11:09:21 Starting syslog-ng: [ OK ] 2014-06-05 11:09:21 Starting ipmi drivers: ipmi device interface 2014-06-05 11:09:21 [ OK ] 2014-06-05 11:09:21 Starting irqbalance: [ OK ] 2014-06-05 11:09:21 Starting rpcbind: [ OK ] 2014-06-05 11:09:21 Starting NFS statd: [ OK ] 2014-06-05 11:09:21 Starting RPC idmapd: [ OK ] 2014-06-05 11:09:22 Starting kdump:[ OK ] 2014-06-05 11:09:22 Starting system message bus: [ OK ] 2014-06-05 11:09:22 Mounting filesystems: [ OK ] 2014-06-05 11:09:22 Starting HAL daemon: [ OK ] 2014-06-05 11:09:22 Starting lm_sensors: not configured, run sensors-detect[WARNING] 2014-06-05 11:09:22 Retrigger failed udev events[ OK ] 2014-06-05 11:09:23 Loading edac DIMM labels: success 2014-06-05 11:09:23 Starting MUNGE: munged[ OK ] 2014-06-05 11:09:23 Starting snmpd: [ OK ] 2014-06-05 11:09:23 Starting xinetd: [ OK ] 2014-06-05 11:09:23 ntpdate: Synchronizing with time server: [ OK ] 2014-06-05 11:09:23 Starting ntpd: [ OK ] 2014-06-05 11:09:23 Releasing lnet ports 2014-06-05 11:09:23 LNet: HW CPU cores: 16, npartitions: 4 2014-06-05 11:09:24 alg: No test for crc32 (crc32-table) 2014-06-05 11:09:24 alg: No test for adler32 (adler32-zlib) 2014-06-05 11:09:24 alg: No test for crc32 (crc32-pclmul) 2014-06-05 11:09:28 padlock: VIA PadLock Hash Engine not detected. 2014-06-05 11:09:29 ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready 2014-06-05 11:09:32 LNet: Added LNI 172.19.1.165@o2ib100 [8/1024/0/180] 2014-06-05 11:09:40 ib0: no IPv6 routers present 2014-06-05 11:10:12 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:10:12 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:10:12 no pools available 2014-06-05 11:10:23 LNET configured 2014-06-05 11:10:23 Starting crond: [ OK ] 2014-06-05 11:10:23 Starting cerebrod: [ OK ] 2014-06-05 11:10:34 SPL: using hostid 0xa8c0cb40 2014-06-05 11:10:58 nodeup: starting at 06/05/14 11:10:58. 2014-06-05 11:10:58 nodeup: starting at 06/05/14 11:10:58. 2014-06-05 11:10:58 nodeup: mounting NFS filesystems 2014-06-05 11:10:58 nodeup: mounting NFS filesystems 2014-06-05 11:10:58 mount_rfs: mounting nfs file systems 2014-06-05 11:10:58 mount_rfs: mounting nfs file systems 2014-06-05 11:10:58 NFS: Cache request denied due to non-unique superblock keys 2014-06-05 11:10:58 mount_rfs: filesystems mounted successfuly 2014-06-05 11:10:58 mount_rfs: filesystems mounted successfuly 2014-06-05 11:10:58 nodeup: mounting Lustre filesystems(If present) 2014-06-05 11:10:58 nodeup: mounting Lustre filesystems(If present) 2014-06-05 11:10:58 mount_rfs: node is not a compute, login, or lustre_client node 2014-06-05 11:10:58 mount_rfs: node is not a compute, login, or lustre_client node 2014-06-05 11:10:58 /: Dropped 178.508M of page cache 2014-06-05 11:10:58 nodeup: complete at 06/05/14 11:10:58. 2014-06-05 11:10:58 nodeup: complete at 06/05/14 11:10:58. 2014-06-05 11:10:58 localrc_startlustre: starting lustre 2014-06-05 11:10:58 Stopping snmpd: [ OK ] 2014-06-05 11:10:59 Shutting down cerebrod: [ OK ] 2014-06-05 11:13:11 Mounting porter-mds1/mgs on /mnt/lustre/local/lse-MGS0000 2014-06-05 11:13:11 Lustre: Lustre: Build Version: 2.4.2-11chaos-11chaos--PRISTINE-2.6.32-431.17.2.1chaos.ch5.2.x86_64 2014-06-05 11:13:13 Mounting porter-mds1/mdt0 on /mnt/lustre/local/lse-MDT0000 2014-06-05 11:13:14 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.120.9@o2ib64 (no target) 2014-06-05 11:13:17 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.120.3@o2ib64 (no target) 2014-06-05 11:13:18 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.120.8@o2ib64 (no target) 2014-06-05 11:13:20 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 172.19.1.90@o2ib100 (no target) 2014-06-05 11:13:20 LustreError: Skipped 1 previous similar message 2014-06-05 11:13:22 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.115.16@o2ib10 (no target) 2014-06-05 11:13:22 LustreError: Skipped 438 previous similar messages 2014-06-05 11:13:28 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.120.23@o2ib7 (no target) 2014-06-05 11:13:28 LustreError: Skipped 1550 previous similar messages 2014-06-05 11:13:36 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.112.241@o2ib6 (no target) 2014-06-05 11:13:36 LustreError: Skipped 719 previous similar messages 2014-06-05 11:13:53 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.120.23@o2ib7 (no target) 2014-06-05 11:13:53 LustreError: Skipped 2867 previous similar messages 2014-06-05 11:14:25 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.113.55@o2ib6 (no target) 2014-06-05 11:14:25 LustreError: Skipped 4439 previous similar messages 2014-06-05 11:14:36 Lustre: lse-MDT0000: Not available for connect from 192.168.116.136@o2ib5 (not set up) 2014-06-05 11:14:36 Lustre: lse-MDT0000: Not available for connect from 192.168.118.209@o2ib5 (not set up) 2014-06-05 11:14:36 Lustre: Skipped 1 previous similar message 2014-06-05 11:14:36 Lustre: Skipped 8 previous similar messages 2014-06-05 11:14:37 Lustre: lse-MDT0000: Not available for connect from 192.168.115.109@o2ib10 (not set up) 2014-06-05 11:14:37 Lustre: Skipped 271 previous similar messages 2014-06-05 11:14:38 Lustre: lse-MDT0000: Not available for connect from 192.168.113.41@o2ib6 (not set up) 2014-06-05 11:14:38 Lustre: Skipped 517 previous similar messages 2014-06-05 11:15:24 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:15:49 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:16:14 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:16:39 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:17:04 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:17:29 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:17:54 INFO: task txg_sync:12140 blocked for more than 120 seconds. 2014-06-05 11:17:54 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:17:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:17:54 txg_sync D 0000000000000009 0 12140 2 0x00000000 2014-06-05 11:17:54 ffff880ff0293ba0 0000000000000046 0000000000000000 0000000000000000 2014-06-05 11:17:54 ffff880ff0293b10 ffffffff81061f32 ffff880ff0293b60 ffffffff810547c9 2014-06-05 11:17:54 ffff8810103d1098 ffff880ff0293fd8 000000000000fbc8 ffff8810103d1098 2014-06-05 11:17:54 Call Trace: 2014-06-05 11:17:54 [<ffffffff81061f32>] ? default_wake_function+0x12/0x20 2014-06-05 11:17:54 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:17:54 [<ffffffff8152b113>] io_schedule+0x73/0xc0 2014-06-05 11:17:54 [<ffffffffa02aeaec>] cv_wait_common+0x8c/0x100 [spl] 2014-06-05 11:17:54 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:17:54 [<ffffffffa02aeb78>] __cv_wait_io+0x18/0x20 [spl] 2014-06-05 11:17:54 [<ffffffffa03da64b>] zio_wait+0xfb/0x1b0 [zfs] 2014-06-05 11:17:54 [<ffffffffa03707b3>] dsl_pool_sync+0xb3/0x3f0 [zfs] 2014-06-05 11:17:54 [<ffffffffa0382d1b>] spa_sync+0x40b/0xa60 [zfs] 2014-06-05 11:17:54 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:17:54 [<ffffffffa0395540>] txg_sync_thread+0x300/0x520 [zfs] 2014-06-05 11:17:54 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:17:54 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:17:54 [<ffffffffa02aa5c8>] thread_generic_wrapper+0x68/0x80 [spl] 2014-06-05 11:17:54 [<ffffffffa02aa560>] ? thread_generic_wrapper+0x0/0x80 [spl] 2014-06-05 11:17:54 [<ffffffff8109b486>] kthread+0x96/0xa0 2014-06-05 11:17:54 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:17:54 [<ffffffff8109b3f0>] ? kthread+0x0/0xa0 2014-06-05 11:17:54 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:17:54 INFO: task ll_mgs_0004:12376 blocked for more than 120 seconds. 2014-06-05 11:17:54 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:17:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:17:54 ll_mgs_0004 D 000000000000000a 0 12376 2 0x00000000 2014-06-05 11:17:54 ffff882009479b20 0000000000000046 0000000000000003 ffff880fe46260c0 2014-06-05 11:17:54 ffff880fe4626068 ffff881ffa46ebe0 ffff882009479ab0 0000000000000001 2014-06-05 11:17:54 ffff88200be40638 ffff882009479fd8 000000000000fbc8 ffff88200be40638 2014-06-05 11:17:54 Call Trace: 2014-06-05 11:17:54 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:17:54 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:17:54 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:17:54 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:17:54 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:17:54 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:17:54 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:17:54 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:17:54 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:17:54 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:17:54 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:17:54 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:17:54 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:17:54 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:17:54 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:17:54 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:17:54 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:17:54 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:17:54 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:17:54 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:17:54 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:17:54 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:17:54 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:18:19 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:18:44 LNet: Service thread pid 12376 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 11:18:44 Pid: 12376, comm: ll_mgs_0004 2014-06-05 11:18:44 2014-06-05 11:18:44 Call Trace: 2014-06-05 11:18:44 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:18:44 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:18:44 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:18:44 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:18:44 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:18:44 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:18:44 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:18:44 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:18:44 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:18:44 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:18:44 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:18:44 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:18:44 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:18:44 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:18:44 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:18:44 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:18:44 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:18:44 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:18:44 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:18:44 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:18:44 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:18:44 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:18:44 2014-06-05 11:18:44 LustreError: dumping log to /tmp/lustre-log.1401992324.12376 2014-06-05 11:19:09 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:19:09 LustreError: Skipped 1 previous similar message 2014-06-05 11:19:54 INFO: task txg_sync:12140 blocked for more than 120 seconds. 2014-06-05 11:19:54 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:19:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:19:54 txg_sync D 0000000000000009 0 12140 2 0x00000000 2014-06-05 11:19:54 ffff880ff0293ba0 0000000000000046 0000000000000000 0000000000000000 2014-06-05 11:19:54 ffff880ff0293b10 ffffffff81061f32 ffff880ff0293b60 ffffffff810547c9 2014-06-05 11:19:54 ffff8810103d1098 ffff880ff0293fd8 000000000000fbc8 ffff8810103d1098 2014-06-05 11:19:54 Call Trace: 2014-06-05 11:19:54 [<ffffffff81061f32>] ? default_wake_function+0x12/0x20 2014-06-05 11:19:54 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:19:54 [<ffffffff8152b113>] io_schedule+0x73/0xc0 2014-06-05 11:19:54 [<ffffffffa02aeaec>] cv_wait_common+0x8c/0x100 [spl] 2014-06-05 11:19:54 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:19:54 [<ffffffffa02aeb78>] __cv_wait_io+0x18/0x20 [spl] 2014-06-05 11:19:54 [<ffffffffa03da64b>] zio_wait+0xfb/0x1b0 [zfs] 2014-06-05 11:19:54 [<ffffffffa03707b3>] dsl_pool_sync+0xb3/0x3f0 [zfs] 2014-06-05 11:19:54 [<ffffffffa0382d1b>] spa_sync+0x40b/0xa60 [zfs] 2014-06-05 11:19:54 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:19:54 [<ffffffffa0395540>] txg_sync_thread+0x300/0x520 [zfs] 2014-06-05 11:19:54 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:19:54 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:19:54 [<ffffffffa02aa5c8>] thread_generic_wrapper+0x68/0x80 [spl] 2014-06-05 11:19:54 [<ffffffffa02aa560>] ? thread_generic_wrapper+0x0/0x80 [spl] 2014-06-05 11:19:54 [<ffffffff8109b486>] kthread+0x96/0xa0 2014-06-05 11:19:54 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:19:54 [<ffffffff8109b3f0>] ? kthread+0x0/0xa0 2014-06-05 11:19:54 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:19:54 INFO: task ll_mgs_0004:12376 blocked for more than 120 seconds. 2014-06-05 11:19:54 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:19:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:19:54 ll_mgs_0004 D 000000000000000a 0 12376 2 0x00000000 2014-06-05 11:19:54 ffff882009479b20 0000000000000046 0000000000000003 ffff880fe46260c0 2014-06-05 11:19:54 ffff880fe4626068 ffff881ffa46ebe0 ffff882009479ab0 0000000000000001 2014-06-05 11:19:54 ffff88200be40638 ffff882009479fd8 000000000000fbc8 ffff88200be40638 2014-06-05 11:19:54 Call Trace: 2014-06-05 11:19:54 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:19:54 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:19:54 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:19:54 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:19:54 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:19:54 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:19:54 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:19:54 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:19:54 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:19:54 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:19:54 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:19:54 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:19:54 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:19:54 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:19:54 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:19:54 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:19:54 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:19:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:19:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:19:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:19:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:19:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:20:24 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:20:24 LustreError: Skipped 2 previous similar messages 2014-06-05 11:21:55 INFO: task txg_sync:12140 blocked for more than 120 seconds. 2014-06-05 11:21:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:21:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:21:55 txg_sync D 0000000000000009 0 12140 2 0x00000000 2014-06-05 11:21:55 ffff880ff0293ba0 0000000000000046 0000000000000000 0000000000000000 2014-06-05 11:21:55 ffff880ff0293b10 ffffffff81061f32 ffff880ff0293b60 ffffffff810547c9 2014-06-05 11:21:55 ffff8810103d1098 ffff880ff0293fd8 000000000000fbc8 ffff8810103d1098 2014-06-05 11:21:55 Call Trace: 2014-06-05 11:21:55 [<ffffffff81061f32>] ? default_wake_function+0x12/0x20 2014-06-05 11:21:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:21:55 [<ffffffff8152b113>] io_schedule+0x73/0xc0 2014-06-05 11:21:55 [<ffffffffa02aeaec>] cv_wait_common+0x8c/0x100 [spl] 2014-06-05 11:21:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:21:55 [<ffffffffa02aeb78>] __cv_wait_io+0x18/0x20 [spl] 2014-06-05 11:21:55 [<ffffffffa03da64b>] zio_wait+0xfb/0x1b0 [zfs] 2014-06-05 11:21:55 [<ffffffffa03707b3>] dsl_pool_sync+0xb3/0x3f0 [zfs] 2014-06-05 11:21:55 [<ffffffffa0382d1b>] spa_sync+0x40b/0xa60 [zfs] 2014-06-05 11:21:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:21:55 [<ffffffffa0395540>] txg_sync_thread+0x300/0x520 [zfs] 2014-06-05 11:21:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:21:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:21:55 [<ffffffffa02aa5c8>] thread_generic_wrapper+0x68/0x80 [spl] 2014-06-05 11:21:55 [<ffffffffa02aa560>] ? thread_generic_wrapper+0x0/0x80 [spl] 2014-06-05 11:21:55 [<ffffffff8109b486>] kthread+0x96/0xa0 2014-06-05 11:21:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:21:55 [<ffffffff8109b3f0>] ? kthread+0x0/0xa0 2014-06-05 11:21:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:21:55 INFO: task ll_mgs_0004:12376 blocked for more than 120 seconds. 2014-06-05 11:21:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:21:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:21:55 ll_mgs_0004 D 000000000000000a 0 12376 2 0x00000000 2014-06-05 11:21:55 ffff882009479b20 0000000000000046 0000000000000003 ffff880fe46260c0 2014-06-05 11:21:55 ffff880fe4626068 ffff881ffa46ebe0 ffff882009479ab0 0000000000000001 2014-06-05 11:21:55 ffff88200be40638 ffff882009479fd8 000000000000fbc8 ffff88200be40638 2014-06-05 11:21:55 Call Trace: 2014-06-05 11:21:55 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:21:55 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:21:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:21:55 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:21:55 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:21:55 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:21:55 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:21:55 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:21:55 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:21:55 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:21:55 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:21:55 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:21:55 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:21:55 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:21:55 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:21:55 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:21:55 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:21:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:21:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:21:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:21:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:21:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:22:54 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:22:54 LustreError: Skipped 5 previous similar messages 2014-06-05 11:23:55 INFO: task txg_sync:12140 blocked for more than 120 seconds. 2014-06-05 11:23:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:23:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:23:55 txg_sync D 0000000000000009 0 12140 2 0x00000000 2014-06-05 11:23:55 ffff880ff0293ba0 0000000000000046 0000000000000000 0000000000000000 2014-06-05 11:23:55 ffff880ff0293b10 ffffffff81061f32 ffff880ff0293b60 ffffffff810547c9 2014-06-05 11:23:55 ffff8810103d1098 ffff880ff0293fd8 000000000000fbc8 ffff8810103d1098 2014-06-05 11:23:55 Call Trace: 2014-06-05 11:23:55 [<ffffffff81061f32>] ? default_wake_function+0x12/0x20 2014-06-05 11:23:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:23:55 [<ffffffff8152b113>] io_schedule+0x73/0xc0 2014-06-05 11:23:55 [<ffffffffa02aeaec>] cv_wait_common+0x8c/0x100 [spl] 2014-06-05 11:23:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:23:55 [<ffffffffa02aeb78>] __cv_wait_io+0x18/0x20 [spl] 2014-06-05 11:23:55 [<ffffffffa03da64b>] zio_wait+0xfb/0x1b0 [zfs] 2014-06-05 11:23:55 [<ffffffffa03707b3>] dsl_pool_sync+0xb3/0x3f0 [zfs] 2014-06-05 11:23:55 [<ffffffffa0382d1b>] spa_sync+0x40b/0xa60 [zfs] 2014-06-05 11:23:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:23:55 [<ffffffffa0395540>] txg_sync_thread+0x300/0x520 [zfs] 2014-06-05 11:23:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:23:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:23:55 [<ffffffffa02aa5c8>] thread_generic_wrapper+0x68/0x80 [spl] 2014-06-05 11:23:55 [<ffffffffa02aa560>] ? thread_generic_wrapper+0x0/0x80 [spl] 2014-06-05 11:23:55 [<ffffffff8109b486>] kthread+0x96/0xa0 2014-06-05 11:23:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:23:55 [<ffffffff8109b3f0>] ? kthread+0x0/0xa0 2014-06-05 11:23:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:23:55 INFO: task ll_mgs_0004:12376 blocked for more than 120 seconds. 2014-06-05 11:23:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:23:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:23:55 ll_mgs_0004 D 000000000000000a 0 12376 2 0x00000000 2014-06-05 11:23:55 ffff882009479b20 0000000000000046 0000000000000003 ffff880fe46260c0 2014-06-05 11:23:55 ffff880fe4626068 ffff881ffa46ebe0 ffff882009479ab0 0000000000000001 2014-06-05 11:23:55 ffff88200be40638 ffff882009479fd8 000000000000fbc8 ffff88200be40638 2014-06-05 11:23:55 Call Trace: 2014-06-05 11:23:55 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:23:55 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:23:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:23:55 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:23:55 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:23:55 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:23:55 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:23:55 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:23:55 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:23:55 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:23:55 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:23:55 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:23:55 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:23:55 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:23:55 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:23:55 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:23:55 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:23:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:23:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:23:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:23:55 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:23:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:25:55 INFO: task txg_sync:12140 blocked for more than 120 seconds. 2014-06-05 11:25:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:25:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:25:55 txg_sync D 0000000000000009 0 12140 2 0x00000000 2014-06-05 11:25:55 ffff880ff0293ba0 0000000000000046 0000000000000000 0000000000000000 2014-06-05 11:25:55 ffff880ff0293b10 ffffffff81061f32 ffff880ff0293b60 ffffffff810547c9 2014-06-05 11:25:55 ffff8810103d1098 ffff880ff0293fd8 000000000000fbc8 ffff8810103d1098 2014-06-05 11:25:55 Call Trace: 2014-06-05 11:25:55 [<ffffffff81061f32>] ? default_wake_function+0x12/0x20 2014-06-05 11:25:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:25:55 [<ffffffff8152b113>] io_schedule+0x73/0xc0 2014-06-05 11:25:55 [<ffffffffa02aeaec>] cv_wait_common+0x8c/0x100 [spl] 2014-06-05 11:25:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:25:55 [<ffffffffa02aeb78>] __cv_wait_io+0x18/0x20 [spl] 2014-06-05 11:25:55 [<ffffffffa03da64b>] zio_wait+0xfb/0x1b0 [zfs] 2014-06-05 11:25:55 [<ffffffffa03707b3>] dsl_pool_sync+0xb3/0x3f0 [zfs] 2014-06-05 11:25:55 [<ffffffffa0382d1b>] spa_sync+0x40b/0xa60 [zfs] 2014-06-05 11:25:55 [<ffffffff810547c9>] ? __wake_up_common+0x59/0x90 2014-06-05 11:25:55 [<ffffffffa0395540>] txg_sync_thread+0x300/0x520 [zfs] 2014-06-05 11:25:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:25:55 [<ffffffffa0395240>] ? txg_sync_thread+0x0/0x520 [zfs] 2014-06-05 11:25:55 [<ffffffffa02aa5c8>] thread_generic_wrapper+0x68/0x80 [spl] 2014-06-05 11:25:55 [<ffffffffa02aa560>] ? thread_generic_wrapper+0x0/0x80 [spl] 2014-06-05 11:25:55 [<ffffffff8109b486>] kthread+0x96/0xa0 2014-06-05 11:25:55 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:25:55 [<ffffffff8109b3f0>] ? kthread+0x0/0xa0 2014-06-05 11:25:55 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:25:55 INFO: task ll_mgs_0004:12376 blocked for more than 120 seconds. 2014-06-05 11:25:55 Tainted: P --------------- 2.6.32-431.17.2.1chaos.ch5.2.x86_64 #1 2014-06-05 11:25:55 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-06-05 11:25:55 ll_mgs_0004 D 000000000000000a 0 12376 2 0x00000000 2014-06-05 11:25:55 ffff882009479b20 0000000000000046 0000000000000003 ffff880fe46260c0 2014-06-05 11:25:55 ffff880fe4626068 ffff881ffa46ebe0 ffff882009479ab0 0000000000000001 2014-06-05 11:25:55 ffff88200be40638 ffff882009479fd8 000000000000fbc8 ffff88200be40638 2014-06-05 11:25:55 Call Trace: 2014-06-05 11:25:55 [<ffffffff8109ba7e>] ? prepare_to_wait_exclusive+0x4e/0x80 2014-06-05 11:25:55 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:25:55 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:25:55 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:25:55 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:25:55 [<ffffffffa0d2e125>] osd_trans_stop+0x395/0x430 [osd_zfs] 2014-06-05 11:25:55 [<ffffffffa0db44b4>] mgs_ir_update+0x244/0xb00 [mgs] 2014-06-05 11:25:55 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 11:25:55 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:25:55 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:25:55 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:25:55 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:25:55 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:25:55 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:25:55 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:25:55 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:25:55 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:25:56 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:25:56 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:25:56 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:25:56 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:25:56 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:27:29 LustreError: 11-0: lse-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. 2014-06-05 11:27:29 LustreError: Skipped 10 previous similar messages 2014-06-05 11:30:48 Lustre: 12375:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-324), not sending early reply 2014-06-05 11:30:48 req@ffff88200be01850 x1470095153955304/t0(0) o253->f2219fc6-0214-0d54-04ba-1a347020048d@0@lo:0/0 lens 4768/0 e 3 to 0 dl 1401993058 ref 2 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 11:33:48 LustreError: 166-1: MGC172.19.1.165@o2ib100: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail 2014-06-05 11:33:48 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:33:48 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:34:01 Lustre: lse-MDT0000: Will be in recovery for at least 5:00, or until 3834 clients reconnect. 2014-06-05 11:34:01 sh: /sys/module/zfs/parameters/zfs_no_write_throttle: No such file or directory 2014-06-05 11:34:01 Starting snmpd: [ OK ] 2014-06-05 11:34:02 Starting cerebrod: [ OK ] 2014-06-05 11:34:02 localrc: running bmc-config.sh (in background) 2014-06-05 11:34:03 2014-06-05 11:34:03 **WARNING**WARNING**WARNING**WARNING**WARNING** 2014-06-05 11:34:03 This is a Department of Energy (DOE) computer system. DOE 2014-06-05 11:34:03 computer systems are provided for the processing of official 2014-06-05 11:34:03 U.S. Government information only. All data contained within 2014-06-05 11:34:03 DOE computer systems is owned by the DOE, and may be audited, 2014-06-05 11:34:03 intercepted, recorded, read, copied, or captured in any 2014-06-05 11:34:03 manner and disclosed in any manner, by authorized personnel. 2014-06-05 11:34:03 THERE IS NO RIGHT OF PRIVACY IN THIS SYSTEM. System personnel 2014-06-05 11:34:03 may disclose any potential evidence of crime found on DOE 2014-06-05 11:34:03 computer systems to appropriate authorities. 2014-06-05 11:34:03 2014-06-05 11:34:03 USE OF THIS SYSTEM BY ANY USER, AUTHORIZED OR UNAUTHORIZED, 2014-06-05 11:34:03 CONSTITUTES CONSENT TO THIS AUDITING, INTERCEPTION, RECORDING, 2014-06-05 11:34:03 READING, COPYING, CAPTURING, AND DISCLOSURE OF COMPUTER ACTIVITY. 2014-06-05 11:34:03 2014-06-05 11:34:03 **WARNING**WARNING**WARNING**WARNING**WARNING** 2014-06-05 11:34:03 2014-06-05 11:34:03 porter-mds1 login: Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:34:13 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:34:16 LustreError: 13356:0:(mdt_open.c:1512:mdt_reint_open()) @@@ [0x2000280e8:0x2db:0x0]/plot_052287->[0x2000233ff:0x2607:0x0] cr_flags=01 mode=0200100000 msg_flag=0x4 not found in open replay. req@ffff881fb77b4c00 x1451002384608516/t0(115984453232) o101->a1bbdbcc-55a6-fa5e-e94b-700b7afc6bb6@172.16.66.58@tcp:0/0 lens 568/3512 e 0 to 0 dl 1401993316 ref 1 fl Interpret:/4/0 rc 0/0 2014-06-05 11:34:16 LustreError: 13356:0:(mdt_open.c:1512:mdt_reint_open()) @@@ [0x2000280e8:0x2db:0x0]/plot_052287->[0x2000233ff:0x2607:0x0] cr_flags=01 mode=0200100000 msg_flag=0x4 not found in open replay. req@ffff880f7c4e2000 x1451002384608512/t0(115984453234) o101->a1bbdbcc-55a6-fa5e-e94b-700b7afc6bb6@172.16.66.58@tcp:0/0 lens 568/3512 e 0 to 0 dl 1401993317 ref 1 fl Interpret:/4/0 rc 0/0 2014-06-05 11:34:38 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:34:38 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:35:03 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:35:03 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:35:11 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:35:11 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:35:28 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:35:28 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:35:53 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:35:53 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:36:18 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:36:18 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:36:18 LustreError: 11-0: MGC172.19.1.165@o2ib100: Communicating with 0@lo, operation mgs_connect failed with -16. 2014-06-05 11:36:18 LustreError: Skipped 21 previous similar messages 2014-06-05 11:36:43 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:36:43 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:37:33 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:37:33 Lustre: Skipped 1 previous similar message 2014-06-05 11:37:33 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:37:33 Lustre: Skipped 1 previous similar message 2014-06-05 11:38:48 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) reconnecting 2014-06-05 11:38:48 Lustre: Skipped 2 previous similar messages 2014-06-05 11:38:48 Lustre: MGS: Client f2219fc6-0214-0d54-04ba-1a347020048d (at 0@lo) refused reconnection, still busy with 1 active RPCs 2014-06-05 11:38:48 Lustre: Skipped 2 previous similar messages <ConMan> Console [porter-mds1] departed by <root@localhost> on pts/2 at 06-05 11:39. 2014-06-05 11:40:01 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3826 recovered, 6 in progress, and 2 evicted) to recover in 1:00 2014-06-05 11:40:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:40:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:40:27 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in 0:35 2014-06-05 11:40:52 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in 0:10 2014-06-05 11:41:17 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in 0:-14 2014-06-05 11:41:42 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in 0:-39 2014-06-05 11:42:07 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -1:-04 2014-06-05 11:42:32 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -1:-29 2014-06-05 11:42:57 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -1:-54 2014-06-05 11:43:47 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -2:-44 2014-06-05 11:43:47 Lustre: Skipped 1 previous similar message 2014-06-05 11:45:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:45:02 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -3:-59 2014-06-05 11:45:02 Lustre: Skipped 2 previous similar messages 2014-06-05 11:45:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:46:43 LustreError: 11-0: lse-OST000a-osc-MDT0000: Communicating with 172.19.1.175@o2ib100, operation ost_connect failed with -19. 2014-06-05 11:46:43 LustreError: Skipped 599 previous similar messages 2014-06-05 11:47:32 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -6:-29 2014-06-05 11:47:32 Lustre: Skipped 5 previous similar messages 2014-06-05 11:49:59 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-41), not sending early reply 2014-06-05 11:49:59 req@ffff880f643cd850 x1451070686103840/t0(0) o400->1fae2579-8440-9b24-00ba-fc52b1a4f32b@192.168.115.67@o2ib10:0/0 lens 224/0 e 1 to 0 dl 1401994209 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:50:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:50:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:50:24 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-41), not sending early reply 2014-06-05 11:50:24 req@ffff880f63beb050 x1451070686104168/t0(0) o400->1fae2579-8440-9b24-00ba-fc52b1a4f32b@192.168.115.67@o2ib10:0/0 lens 224/0 e 1 to 0 dl 1401994234 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:51:13 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:13 req@ffff88200bdf2850 x1470096765616136/t0(0) o253->ba4d35b8-5dca-547f-e76a-1aa10c9f3093@172.19.1.167@o2ib100:0/0 lens 4768/4768 e 0 to 0 dl 1401994283 ref 2 fl Interpret:/0/0 rc 0/0 2014-06-05 11:51:16 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:16 req@ffff88200bdfe050 x1470096768761864/t0(0) o253->1c1864a2-adcc-54be-61ba-1841d8e9254f@172.19.1.171@o2ib100:0/0 lens 4768/4768 e 0 to 0 dl 1401994286 ref 2 fl Interpret:/0/0 rc 0/0 2014-06-05 11:51:16 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message 2014-06-05 11:51:19 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:19 req@ffff88200be0f050 x1470096770859016/t0(0) o253->c1f6f54b-f50b-93c5-6f86-2cfaf6dd0951@172.19.1.174@o2ib100:0/0 lens 4768/4768 e 0 to 0 dl 1401994289 ref 2 fl Interpret:/0/0 rc 0/0 2014-06-05 11:51:19 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages 2014-06-05 11:51:24 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:24 req@ffff88200be14050 x1470096776101896/t0(0) o253->dbfc3de4-c8fb-99ca-101e-df0293d2b241@172.19.1.180@o2ib100:0/0 lens 4768/4768 e 0 to 0 dl 1401994294 ref 2 fl Interpret:/0/0 rc 0/0 2014-06-05 11:51:24 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages 2014-06-05 11:51:33 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:33 req@ffff880f645b0050 x1467564917545724/t0(0) o400->adb8b76b-93c2-16a5-8fb4-37d0543cf7d7@192.168.120.8@o2ib64:0/0 lens 224/0 e 0 to 0 dl 1401994303 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:51:33 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 244 previous similar messages 2014-06-05 11:51:50 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:51:50 req@ffff880f63d5e050 x1451072354109904/t0(0) o400->00fa4106-0286-54ed-a379-60de99ba8226@192.168.114.15@o2ib10:0/0 lens 224/0 e 0 to 0 dl 1401994320 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:51:50 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 3364 previous similar messages 2014-06-05 11:52:07 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -11:-04 2014-06-05 11:52:07 Lustre: Skipped 10 previous similar messages 2014-06-05 11:52:23 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:52:23 req@ffff880f630aa850 x1467564917546244/t0(0) o400->adb8b76b-93c2-16a5-8fb4-37d0543cf7d7@192.168.120.8@o2ib64:0/0 lens 224/0 e 0 to 0 dl 1401994353 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:52:23 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 4326 previous similar messages 2014-06-05 11:53:28 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:53:28 req@ffff880f59ef2050 x1457254531904460/t0(0) o400->138c0cc8-b8d3-134a-dd43-d5111d33a590@192.168.115.145@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1401994418 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:53:28 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 10825 previous similar messages 2014-06-05 11:55:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:55:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 11:55:37 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:55:37 req@ffff880f533d0050 x1451072285344932/t0(0) o400->fbae1215-0372-9e68-d8ff-54632336370a@192.168.117.123@o2ib10:0/0 lens 224/0 e 0 to 0 dl 1401994547 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:55:37 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 19893 previous similar messages 2014-06-05 11:55:55 Lustre: MGS: haven't heard from client ba4d35b8-5dca-547f-e76a-1aa10c9f3093 (at 172.19.1.167@o2ib100) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6815b000, cur 1401994555 expire 1401994405 last 1401994328 2014-06-05 11:55:55 Lustre: MGS: haven't heard from client 23b53c12-997a-20fc-c582-9c26628e5706 (at 172.19.1.168@o2ib100) in 226 seconds. I think it's dead, and I am evicting it. exp ffff880f6d389000, cur 1401994555 expire 1401994405 last 1401994329 2014-06-05 11:57:11 Lustre: MGS: haven't heard from client ff32d74b-d942-895c-457b-1b01d40bf4dd (at 192.168.120.6@o2ib7) in 154 seconds. I think it's dead, and I am evicting it. exp ffff880fdde4bc00, cur 1401994631 expire 1401994481 last 1401994477 2014-06-05 11:57:11 Lustre: Skipped 28 previous similar messages 2014-06-05 11:57:33 LustreError: 11-0: lse-OST0011-osc-MDT0000: Communicating with 172.19.1.184@o2ib100, operation ost_connect failed with -19. 2014-06-05 11:57:33 LustreError: Skipped 444 previous similar messages 2014-06-05 11:58:27 Lustre: MGS: haven't heard from client d01d5237-1aa5-287a-ac4e-836be2aa9d12 (at 192.168.120.207@o2ib7) in 225 seconds. I think it's dead, and I am evicting it. exp ffff8820096f9800, cur 1401994707 expire 1401994557 last 1401994482 2014-06-05 11:58:27 Lustre: Skipped 4 previous similar messages 2014-06-05 11:58:48 LNet: Service thread pid 12306 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 11:58:48 Pid: 12306, comm: ll_mgs_0001 2014-06-05 11:58:48 2014-06-05 11:58:48 Call Trace: 2014-06-05 11:58:48 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:58:48 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:58:48 [<ffffffffa08c9e65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-06-05 11:58:48 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:58:48 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:58:48 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 11:58:48 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 11:58:48 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:58:48 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:58:48 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:58:48 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:58:48 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:58:48 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:58:48 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:58:48 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:58:48 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:58:48 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:48 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:58:48 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:48 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:48 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:58:48 2014-06-05 11:58:48 LustreError: dumping log to /tmp/lustre-log.1401994728.12306 2014-06-05 11:58:49 LNet: Service thread pid 12375 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 11:58:49 Pid: 12375, comm: ll_mgs_0003 2014-06-05 11:58:49 2014-06-05 11:58:49 Call Trace: 2014-06-05 11:58:49 [<ffffffff81061d8e>] ? try_to_wake_up+0x24e/0x3e0 2014-06-05 11:58:49 [<ffffffff81061f75>] ? wake_up_process+0x15/0x20 2014-06-05 11:58:49 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:58:49 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:58:49 [<ffffffffa08c9e65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-06-05 11:58:49 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:58:49 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:58:49 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 11:58:49 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 11:58:49 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:58:49 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:58:49 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:58:49 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:58:49 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:58:49 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:58:49 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:58:49 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:58:49 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:58:49 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:49 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:58:49 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:49 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:49 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:58:49 2014-06-05 11:58:49 LustreError: dumping log to /tmp/lustre-log.1401994729.12375 2014-06-05 11:58:51 LNet: Service thread pid 12381 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 11:58:51 Pid: 12381, comm: ll_mgs_0005 2014-06-05 11:58:51 2014-06-05 11:58:51 Call Trace: 2014-06-05 11:58:51 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:58:51 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:58:51 [<ffffffffa08c9e65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-06-05 11:58:51 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:58:51 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:58:51 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 11:58:51 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 11:58:51 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:58:51 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:58:51 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:58:51 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:58:51 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:58:51 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:58:51 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:58:51 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:58:51 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:58:51 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:51 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:58:51 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:51 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:51 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:58:51 2014-06-05 11:58:51 LustreError: dumping log to /tmp/lustre-log.1401994731.12381 2014-06-05 11:58:52 Pid: 12307, comm: ll_mgs_0002 2014-06-05 11:58:52 2014-06-05 11:58:52 Call Trace: 2014-06-05 11:58:52 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:58:52 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:58:52 [<ffffffffa08c9e65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:58:52 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:58:52 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 11:58:52 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 11:58:52 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:58:52 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:58:52 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:58:52 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:58:52 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:58:52 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:58:52 2014-06-05 11:58:52 LustreError: dumping log to /tmp/lustre-log.1401994732.12307 2014-06-05 11:58:52 LNet: Service thread pid 13505 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 11:58:52 LNet: Skipped 1 previous similar message 2014-06-05 11:58:52 Pid: 13505, comm: ll_mgs_0006 2014-06-05 11:58:52 2014-06-05 11:58:52 Call Trace: 2014-06-05 11:58:52 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 11:58:52 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 11:58:52 [<ffffffffa08c9e65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 11:58:52 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 11:58:52 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 11:58:52 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 11:58:52 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 11:58:52 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 11:58:52 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 11:58:52 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 11:58:52 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 11:58:52 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 11:58:52 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 11:58:52 2014-06-05 11:58:52 LustreError: dumping log to /tmp/lustre-log.1401994732.13505 2014-06-05 11:58:53 LNet: Service thread pid 13506 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:58:53 LustreError: dumping log to /tmp/lustre-log.1401994733.13506 2014-06-05 11:58:53 LNet: Service thread pid 13507 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:58:53 LustreError: dumping log to /tmp/lustre-log.1401994733.13507 2014-06-05 11:58:53 LustreError: dumping log to /tmp/lustre-log.1401994733.13508 2014-06-05 11:58:53 LustreError: dumping log to /tmp/lustre-log.1401994733.13509 2014-06-05 11:58:54 LNet: Service thread pid 13510 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:58:54 LNet: Skipped 2 previous similar messages 2014-06-05 11:58:54 LustreError: dumping log to /tmp/lustre-log.1401994734.13510 2014-06-05 11:58:55 LNet: Service thread pid 13511 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:58:55 LustreError: dumping log to /tmp/lustre-log.1401994735.13511 2014-06-05 11:58:55 LustreError: dumping log to /tmp/lustre-log.1401994735.13512 2014-06-05 11:58:55 LustreError: dumping log to /tmp/lustre-log.1401994735.13513 2014-06-05 11:58:55 LustreError: dumping log to /tmp/lustre-log.1401994735.13514 2014-06-05 11:58:57 LustreError: dumping log to /tmp/lustre-log.1401994737.13515 2014-06-05 11:58:57 LNet: Service thread pid 13516 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:58:57 LNet: Skipped 4 previous similar messages 2014-06-05 11:58:57 LustreError: dumping log to /tmp/lustre-log.1401994737.13516 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13517 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13518 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13519 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13520 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13521 2014-06-05 11:58:58 LustreError: dumping log to /tmp/lustre-log.1401994738.13522 2014-06-05 11:58:59 LustreError: dumping log to /tmp/lustre-log.1401994738.13523 2014-06-05 11:58:59 LustreError: dumping log to /tmp/lustre-log.1401994739.13524 2014-06-05 11:59:00 LustreError: dumping log to /tmp/lustre-log.1401994740.13525 2014-06-05 11:59:00 LustreError: dumping log to /tmp/lustre-log.1401994740.13526 2014-06-05 11:59:00 LustreError: dumping log to /tmp/lustre-log.1401994740.13527 2014-06-05 11:59:01 LustreError: dumping log to /tmp/lustre-log.1401994740.13528 2014-06-05 11:59:01 LustreError: dumping log to /tmp/lustre-log.1401994741.13530 2014-06-05 11:59:01 LNet: Service thread pid 13529 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 11:59:01 LNet: Skipped 13 previous similar messages 2014-06-05 11:59:01 LustreError: dumping log to /tmp/lustre-log.1401994741.13529 2014-06-05 11:59:46 Lustre: MGS: haven't heard from client 2923aa2a-84ba-38b4-d8de-f42adfa58309 (at 192.168.120.166@o2ib7) in 229 seconds. I think it's dead, and I am evicting it. exp ffff882009186800, cur 1401994786 expire 1401994636 last 1401994557 2014-06-05 11:59:46 Lustre: Skipped 150 previous similar messages 2014-06-05 11:59:54 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 11:59:54 req@ffff880f465bb050 x1459408878318748/t0(0) o400->e1af2157-ae6e-d9cc-4621-106974b0eafb@192.168.120.104@o2ib7:0/0 lens 224/0 e 0 to 0 dl 1401994804 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 11:59:54 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 38786 previous similar messages <ConMan> Console [porter-mds1] log at 2014-06-05 12:00:00 PDT. 2014-06-05 12:00:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:00:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:00:53 Lustre: lse-MDT0000: Denying connection for new client 2a88b03e-db10-e85f-ce3e-03067d970d80 (at 172.19.1.85@o2ib100), waiting for all 3834 known clients (3832 recovered, 0 in progress, and 2 evicted) to recover in -19:-50 2014-06-05 12:00:53 Lustre: Skipped 20 previous similar messages 2014-06-05 12:05:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:05:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:05:58 Lustre: lse-MDT0000: Recovery over after 31:57, of 3834 clients 3832 recovered and 2 were evicted. 2014-06-05 12:06:04 Lustre: lse-MDT0000: Recovery exceed expected maximum time of 5:00. 2014-06-05 12:06:04 LustreError: dumping log to /tmp/lustre-log.1401995164.12822 2014-06-05 12:07:14 Lustre: lse-MDT0000: haven't heard from client e9718db3-7ce2-f7b7-d4c3-b5833653d11e (at 192.168.120.52@o2ib7) in 1992 seconds. I think it's dead, and I am evicting it. exp ffff882003231400, cur 1401995234 expire 1401995084 last 1401993242 2014-06-05 12:07:14 Lustre: Skipped 4 previous similar messages 2014-06-05 12:07:58 LustreError: 11-0: lse-OST0003-osc-MDT0000: Communicating with 172.19.1.169@o2ib100, operation ost_connect failed with -19. 2014-06-05 12:07:58 LustreError: Skipped 40 previous similar messages 2014-06-05 12:08:29 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 0@lo ns: MGS lock: ffff880fd3e16d80/0x49bd4c25c32c31a5 lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3674 type: PLN flags: 0x20 nid: 0@lo remote: 0x49bd4c25c32c319e expref: 33 pid: 12381 timeout: 4298303310 lvb_type: 0 2014-06-05 12:08:29 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 192.168.120.24@o2ib14 ns: MGS lock: ffff88200676e6c0/0x49bd4c25c32bbf7c lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3674 type: PLN flags: 0x20 nid: 192.168.120.24@o2ib14 remote: 0xdb86fff815ee2688 expref: 53 pid: 12305 timeout: 4298303310 lvb_type: 0 2014-06-05 12:08:42 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 12:08:42 req@ffff880f34d8b850 x1461695209068696/t0(0) o400->9d7cbf6e-39af-f906-dc3b-85dbef501011@172.16.66.66@tcp:0/0 lens 224/0 e 0 to 0 dl 1401995332 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 12:08:42 Lustre: 12305:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 47951 previous similar messages 2014-06-05 12:10:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:10:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:15:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:15:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:17:58 LustreError: 11-0: lse-OST000f-osc-MDT0000: Communicating with 172.19.1.182@o2ib100, operation ost_connect failed with -19. 2014-06-05 12:17:58 LustreError: Skipped 56 previous similar messages 2014-06-05 12:20:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:20:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:22:42 Lustre: lse-MDT0000: haven't heard from client af4ab5c3-8099-81ac-b3e3-1e95c333219d (at 192.168.118.20@o2ib10) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820029db000, cur 1401996162 expire 1401996012 last 1401995935 2014-06-05 12:22:42 Lustre: Skipped 160 previous similar messages 2014-06-05 12:25:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:25:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:28:23 LustreError: 11-0: lse-OST001b-osc-MDT0000: Communicating with 172.19.1.194@o2ib100, operation ost_connect failed with -19. 2014-06-05 12:28:23 LustreError: Skipped 67 previous similar messages 2014-06-05 12:30:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:30:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:35:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:35:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:36:31 ----------- 2014-06-05 12:38:48 LustreError: 11-0: lse-OST000c-osc-MDT0000: Communicating with 172.19.1.178@o2ib100, operation ost_connect failed with -19. 2014-06-05 12:38:48 LustreError: Skipped 39 previous similar messages 2014-06-05 12:40:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:40:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:45:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:45:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:49:38 LustreError: 11-0: lse-OST003a-osc-MDT0000: Communicating with 172.19.1.226@o2ib100, operation ost_connect failed with -19. 2014-06-05 12:49:38 LustreError: Skipped 45 previous similar messages 2014-06-05 12:50:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:50:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:55:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:55:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 12:55:11 Lustre: 13515:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3819s); client may timeout. req@ffff88200be0e850 x1470096774004744/t0(0) o253->8ffe4564-b4f8-3661-7010-61acd352b376@172.19.1.187@o2ib100:0/0 lens 4768/4736 e 0 to 0 dl 1401994292 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 12:55:11 Lustre: 13528:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3816s); client may timeout. req@ffff88200be0c850 x1470096778199048/t0(0) o253->34fb902b-eccb-3297-451c-f60a6929e976@172.19.1.204@o2ib100:0/0 lens 4768/4736 e 0 to 0 dl 1401994295 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 12:55:11 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3818s); client may timeout. req@ffff88200be07050 x1470096776101896/t0(0) o253->3449aa52-8dc7-70b6-fd27-d52c1115eedf@172.19.1.202@o2ib100:0/0 lens 4768/4736 e 0 to 0 dl 1401994293 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 12:55:11 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 1 previous similar message 2014-06-05 12:55:11 LNet: Service thread pid 13528 completed after 4570.42s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 12:55:11 LNet: Skipped 1 previous similar message 2014-06-05 12:55:11 LustreError: 13509:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.112.19@o2ib6: deadline 755:3815s ago 2014-06-05 12:55:11 req@ffff88200be08850 x1459135880782520/t0(0) o400->98e70509-77b1-c00c-e060-c62216922c2a@192.168.112.19@o2ib6:0/0 lens 224/0 e 0 to 0 dl 1401994296 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:11 LustreError: 13509:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 2 previous similar messages 2014-06-05 12:55:11 LNet: 10964:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.202@o2ib100 2014-06-05 12:55:11 Lustre: 13526:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3805s); client may timeout. req@ffff880f64685850 x1467388024436332/t0(0) o400->206f4df6-bf73-61b2-deea-72568ee353a1@192.168.116.25@o2ib10:0/0 lens 224/0 e 0 to 0 dl 1401994306 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:11 LustreError: 13523:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.115.208@o2ib5: deadline 755:3805s ago 2014-06-05 12:55:11 req@ffff880f64685050 x1467850563960500/t0(0) o400->d14471fe-8c33-de37-3a83-fb0ea2486fb5@192.168.115.208@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1401994306 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:11 Lustre: 13526:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 645 previous similar messages 2014-06-05 12:55:11 LustreError: 13523:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 643 previous similar messages 2014-06-05 12:55:11 LNet: 10968:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.177@o2ib100 2014-06-05 12:55:11 LNet: 10968:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) Skipped 6 previous similar messages 2014-06-05 12:55:11 LNet: Service thread pid 13517 completed after 4573.29s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 12:55:11 LNet: Skipped 10 previous similar messages 2014-06-05 12:55:11 Lustre: 13515:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 112628 previous similar messages 2014-06-05 12:55:11 LustreError: 13520:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.116.54@o2ib10: deadline 100:2470s ago 2014-06-05 12:55:11 req@ffff880ef73d1850 x1451072333718684/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1401995641 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:11 Lustre: 12375:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:2470s); client may timeout. req@ffff880ef73d1050 x1458679643095124/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1401995641 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:12 Lustre: 13510:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:2470s); client may timeout. req@ffff880ef73d0850 x1450902320057396/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1401995641 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 12:55:12 Lustre: 12375:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 66504 previous similar messages 2014-06-05 12:55:12 Lustre: 13510:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 66507 previous similar messages 2014-06-05 12:55:12 LustreError: 13520:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 216211 previous similar messages 2014-06-05 12:55:12 Lustre: MGS: Client 0779f113-13a4-a30f-cde9-167c110307d2 (at 192.168.115.43@o2ib5) reconnecting 2014-06-05 12:55:12 Lustre: Skipped 2279 previous similar messages 2014-06-05 12:58:59 Lustre: MGS: haven't heard from client f049c034-c21e-9709-de7a-0722f984c024 (at 172.19.1.227@o2ib100) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd72da400, cur 1401998339 expire 1401998189 last 1401998112 <ConMan> Console [porter-mds1] log at 2014-06-05 13:00:00 PDT. 2014-06-05 13:00:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:00:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:00:03 LustreError: 11-0: lse-OST0013-osc-MDT0000: Communicating with 172.19.1.185@o2ib100, operation ost_connect failed with -11. 2014-06-05 13:00:03 LustreError: Skipped 145 previous similar messages 2014-06-05 13:05:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:05:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:05:59 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-41), not sending early reply 2014-06-05 13:05:59 req@ffff880f646d8850 x1451070686169040/t0(0) o101->1fae2579-8440-9b24-00ba-fc52b1a4f32b@192.168.115.67@o2ib10:0/0 lens 328/0 e 1 to 0 dl 1401998769 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 13:05:59 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages 2014-06-05 13:07:36 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-41), not sending early reply 2014-06-05 13:07:36 req@ffff880f5cf7d050 x1451070686170024/t0(0) o400->1fae2579-8440-9b24-00ba-fc52b1a4f32b@192.168.115.67@o2ib10:0/0 lens 224/0 e 1 to 0 dl 1401998866 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 13:07:36 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages 2014-06-05 13:10:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:10:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:10:03 LustreError: 11-0: lse-OST0009-osc-MDT0000: Communicating with 172.19.1.176@o2ib100, operation ost_connect failed with -19. 2014-06-05 13:10:03 LustreError: Skipped 257 previous similar messages 2014-06-05 13:10:06 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 13:10:06 req@ffff880f583c3050 x1465651622559236/t0(0) o400->e63c44a9-6086-0b5a-c17f-58f0d76cf8ed@172.16.66.61@tcp:0/0 lens 224/0 e 0 to 0 dl 1401999016 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 13:10:06 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 21181 previous similar messages 2014-06-05 13:12:23 Lustre: MGS: haven't heard from client f5a80352-0c76-40de-99af-ea6d70654464 (at 172.19.1.246@o2ib100) in 229 seconds. I think it's dead, and I am evicting it. exp ffff881d7a67d400, cur 1401999143 expire 1401998993 last 1401998914 2014-06-05 13:12:23 Lustre: Skipped 22 previous similar messages 2014-06-05 13:12:23 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 13:12:23 Lustre: MGS: haven't heard from client ec5fa9d3-0d7f-af2a-9d07-8dcc3f6052a6 (at 172.19.1.175@o2ib100) in 228 seconds. I think it's dead, and I am evicting it. exp ffff881d01894800, cur 1401999143 expire 1401998993 last 1401998915 2014-06-05 13:15:03 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:15:04 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:15:06 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 13:15:06 req@ffff880f47f68050 x1465651622563380/t0(0) o400->e63c44a9-6086-0b5a-c17f-58f0d76cf8ed@172.16.66.61@tcp:0/0 lens 224/0 e 0 to 0 dl 1401999316 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 13:15:06 Lustre: 13523:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 44064 previous similar messages 2014-06-05 13:15:12 LNet: Service thread pid 13507 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 13:15:12 Pid: 13507, comm: ll_mgs_0008 2014-06-05 13:15:12 2014-06-05 13:15:12 Call Trace: 2014-06-05 13:15:12 [<ffffffffa05dadbb>] ? libcfs_debug_vmsg2+0x50b/0xbb0 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08ccd74>] ? _debug_req+0x454/0x680 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8152c13e>] __mutex_lock_slowpath+0x13e/0x180 2014-06-05 13:15:12 [<ffffffff8152bfde>] mutex_lock+0x3e/0x60 2014-06-05 13:15:12 [<ffffffffa0db4316>] mgs_ir_update+0xa6/0xb00 [mgs] 2014-06-05 13:15:12 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:15:12 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 13:15:12 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:15:12 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 13:15:12 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 13:15:12 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 13:15:12 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:15:12 2014-06-05 13:15:12 LustreError: dumping log to /tmp/lustre-log.1401999312.13507 2014-06-05 13:15:12 Pid: 13510, comm: ll_mgs_0011 2014-06-05 13:15:12 2014-06-05 13:15:12 Call Trace: 2014-06-05 13:15:12 [<ffffffffa05dadbb>] ? libcfs_debug_vmsg2+0x50b/0xbb0 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08ccd74>] ? _debug_req+0x454/0x680 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8152c13e>] __mutex_lock_slowpath+0x13e/0x180 2014-06-05 13:15:12 [<ffffffff8152bfde>] mutex_lock+0x3e/0x60 2014-06-05 13:15:12 [<ffffffffa0db4316>] mgs_ir_update+0xa6/0xb00 [mgs] 2014-06-05 13:15:12 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:15:12 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 13:15:12 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:15:12 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 13:15:12 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 13:15:12 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 13:15:12 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:15:12 2014-06-05 13:15:12 LNet: Service thread pid 13517 was inactive for 1200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 13:15:12 LNet: Skipped 1 previous similar message 2014-06-05 13:15:12 Pid: 13517, comm: ll_mgs_0018 2014-06-05 13:15:12 2014-06-05 13:15:12 Call Trace: 2014-06-05 13:15:12 [<ffffffff8152c13e>] __mutex_lock_slowpath+0x13e/0x180 2014-06-05 13:15:12 [<ffffffff8152bfde>] mutex_lock+0x3e/0x60 2014-06-05 13:15:12 [<ffffffffa0db4316>] mgs_ir_update+0xa6/0xb00 [mgs] 2014-06-05 13:15:12 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:15:12 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 13:15:12 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:15:12 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 13:15:12 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 13:15:12 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 13:15:12 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:15:12 2014-06-05 13:15:12 Pid: 13525, comm: ll_mgs_0026 2014-06-05 13:15:12 2014-06-05 13:15:12 Call Trace: 2014-06-05 13:15:12 [<ffffffffa05dadbb>] ? libcfs_debug_vmsg2+0x50b/0xbb0 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08ccd74>] ? _debug_req+0x454/0x680 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff8152c13e>] __mutex_lock_slowpath+0x13e/0x180 2014-06-05 13:15:12 [<ffffffff8152bfde>] mutex_lock+0x3e/0x60 2014-06-05 13:15:12 [<ffffffffa0db4316>] mgs_ir_update+0xa6/0xb00 [mgs] 2014-06-05 13:15:12 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:15:12 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 13:15:12 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:15:12 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 13:15:12 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:15:12 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 13:15:12 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 13:15:12 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:15:12 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 13:15:12 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:15:13 2014-06-05 13:15:13 Pid: 13521, comm: ll_mgs_0022 2014-06-05 13:15:13 2014-06-05 13:15:13 Call Trace: 2014-06-05 13:15:13 [<ffffffffa05dadbb>] ? libcfs_debug_vmsg2+0x50b/0xbb0 [libcfs] 2014-06-05 13:15:13 [<ffffffffa08ccd74>] ? _debug_req+0x454/0x680 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff8152c13e>] __mutex_lock_slowpath+0x13e/0x180 2014-06-05 13:15:13 [<ffffffff8152bfde>] mutex_lock+0x3e/0x60 2014-06-05 13:15:13 [<ffffffffa0db4316>] mgs_ir_update+0xa6/0xb00 [mgs] 2014-06-05 13:15:13 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:15:13 [<ffffffffa0d9287c>] mgs_handle_target_reg+0x40c/0xe30 [mgs] 2014-06-05 13:15:13 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:15:13 [<ffffffffa073d64f>] ? keys_fill+0x6f/0x190 [obdclass] 2014-06-05 13:15:13 [<ffffffffa08cb8dc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa05cb63e>] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-06-05 13:15:13 [<ffffffffa05dcf6f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-06-05 13:15:13 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff81058ce3>] ? __wake_up+0x53/0x70 2014-06-05 13:15:13 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:15:13 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:15:13 2014-06-05 13:15:13 LNet: Service thread pid 13509 was inactive for 1200.70s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 13:15:14 LNet: Service thread pid 13512 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 13:15:14 LNet: Skipped 1 previous similar message 2014-06-05 13:15:14 LustreError: dumping log to /tmp/lustre-log.1401999314.13512 2014-06-05 13:15:16 LustreError: dumping log to /tmp/lustre-log.1401999315.12307 2014-06-05 13:15:17 LNet: Service thread pid 13520 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. 2014-06-05 13:15:17 LNet: Skipped 1 previous similar message 2014-06-05 13:15:17 LustreError: dumping log to /tmp/lustre-log.1401999317.13520 2014-06-05 13:15:17 LustreError: dumping log to /tmp/lustre-log.1401999317.13527 2014-06-05 13:19:22 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (934:6504s); client may timeout. req@ffff88200be01850 x1470095153955304/t0(0) o253->f2219fc6-0214-0d54-04ba-1a347020048d@0@lo:0/0 lens 4768/4736 e 3 to 0 dl 1401993058 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 13:19:22 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 119211 previous similar messages 2014-06-05 13:19:22 LNet: Service thread pid 12376 completed after 7437.55s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 13:19:22 LNet: Skipped 17 previous similar messages 2014-06-05 13:19:22 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.117.35@o2ib5: deadline 755:690s ago 2014-06-05 13:19:22 req@ffff880f64730050 x1467850812696856/t0(0) o101->400b9504-7163-7c9b-8218-5b292b3f45d9@192.168.117.35@o2ib5:0/0 lens 328/0 e 0 to 0 dl 1401998872 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:22 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 83520 previous similar messages 2014-06-05 13:19:22 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.112.161@o2ib6: deadline 755:686s ago 2014-06-05 13:19:22 req@ffff880f63cce850 x1459135982660436/t0(0) o101->9427781b-dde8-5756-3869-0f0d3ceb960f@192.168.112.161@o2ib6:0/0 lens 328/0 e 0 to 0 dl 1401998876 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:22 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 2625 previous similar messages 2014-06-05 13:19:22 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:686s); client may timeout. req@ffff880f63dbe850 x1459135981903592/t0(0) o101->7f535064-2c99-dbd5-25b2-de1317bcd99f@192.168.113.57@o2ib6:0/0 lens 328/0 e 0 to 0 dl 1401998876 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:22 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 3101 previous similar messages 2014-06-05 13:19:23 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.112.177@o2ib6: deadline 755:577s ago 2014-06-05 13:19:23 req@ffff880f74a8b850 x1459136018247944/t0(0) o400->b028d5f3-98b2-3a98-8c16-02883500bad9@192.168.112.177@o2ib6:0/0 lens 224/0 e 0 to 0 dl 1401998985 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:23 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 14458 previous similar messages 2014-06-05 13:19:23 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:571s); client may timeout. req@ffff880f5a457050 x1465124814264780/t0(0) o400->013aa102-879d-e834-bde6-d6cbfe32468e@192.168.114.246@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1401998992 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:23 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 14611 previous similar messages 2014-06-05 13:19:24 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.112.197@o2ib10: deadline 755:342s ago 2014-06-05 13:19:24 req@ffff880f4fb48850 x1458679461598804/t0(0) o400->2726edc9-28cc-53d7-fc08-84ec0d064d6b@192.168.112.197@o2ib10:0/0 lens 224/0 e 0 to 0 dl 1401999221 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:24 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 34380 previous similar messages 2014-06-05 13:19:24 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:333s); client may timeout. req@ffff880f4f648050 x1467849728339852/t0(0) o400->e5e24355-c8ba-fe1e-a08f-884e7f2305fe@192.168.113.224@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1401999231 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:24 Lustre: 12376:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 34385 previous similar messages 2014-06-05 13:19:26 LustreError: 13523:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.113.53@o2ib5: deadline 100:598s ago 2014-06-05 13:19:26 req@ffff880f3e31b050 x1467849719096748/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1401998967 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:26 LustreError: 13523:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 50790 previous similar messages 2014-06-05 13:19:26 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:589s); client may timeout. req@ffff880f43822850 x1470096774005300/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1401998977 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:19:26 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 50164 previous similar messages 2014-06-05 13:19:27 Lustre: MGS: Client 1fae2579-8440-9b24-00ba-fc52b1a4f32b (at 192.168.115.67@o2ib10) reconnecting 2014-06-05 13:19:27 Lustre: Skipped 409 previous similar messages 2014-06-05 13:19:27 Lustre: MGS: Client fcc2b94c-1611-2aab-a486-02aa236b124a (at 172.19.1.198@o2ib100) refused reconnection, still busy with 1 active RPCs 2014-06-05 13:19:27 Lustre: Evicted from MGS (at MGC172.19.1.165@o2ib100_0) after server handle changed from 0x49bd4c25c329b378 to 0x49bd4c25c3fe7c44 2014-06-05 13:19:27 Lustre: MGC172.19.1.165@o2ib100: Connection restored to MGS (at 0@lo) 2014-06-05 13:20:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:20:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:20:03 LustreError: 11-0: lse-OST000d-osc-MDT0000: Communicating with 172.19.1.179@o2ib100, operation ost_connect failed with -19. 2014-06-05 13:20:03 LustreError: Skipped 254 previous similar messages 2014-06-05 13:25:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:25:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:26:59 Lustre: lse-MDT0000: haven't heard from client ef470bba-b267-ff69-5286-c0875c7e3918 (at 172.16.65.9@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882003070000, cur 1402000019 expire 1401999869 last 1401999792 2014-06-05 13:30:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:30:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:30:03 LustreError: 11-0: lse-OST000d-osc-MDT0000: Communicating with 172.19.1.179@o2ib100, operation ost_connect failed with -19. 2014-06-05 13:30:03 LustreError: Skipped 312 previous similar messages 2014-06-05 13:30:08 Lustre: 12376:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-41), not sending early reply 2014-06-05 13:30:08 req@ffff880f1eba2850 x1470096776102520/t0(0) o253->bea6686c-8f05-839a-09e9-87eef3aabd19@172.19.1.179@o2ib100:0/0 lens 4768/4768 e 1 to 0 dl 1402000218 ref 2 fl Interpret:/0/0 rc 0/0 2014-06-05 13:30:08 Lustre: 12376:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 37723 previous similar messages 2014-06-05 13:32:56 LustreError: 166-1: MGC172.19.1.165@o2ib100: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail 2014-06-05 13:35:03 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:35:04 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:39:27 LNet: Service thread pid 13523 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 2014-06-05 13:39:27 LNet: Skipped 2 previous similar messages 2014-06-05 13:39:27 Pid: 13523, comm: ll_mgs_0024 2014-06-05 13:39:27 2014-06-05 13:39:27 Call Trace: 2014-06-05 13:39:27 [<ffffffffa02aeb4d>] cv_wait_common+0xed/0x100 [spl] 2014-06-05 13:39:27 [<ffffffff8109b830>] ? autoremove_wake_function+0x0/0x40 2014-06-05 13:39:27 [<ffffffffa02aebb5>] __cv_wait+0x15/0x20 [spl] 2014-06-05 13:39:27 [<ffffffffa0394ebb>] txg_wait_synced+0x7b/0xa0 [zfs] 2014-06-05 13:39:27 [<ffffffffa0d2db10>] osd_sync+0xe0/0xf0 [osd_zfs] 2014-06-05 13:39:27 [<ffffffffa0d9297e>] mgs_handle_target_reg+0x50e/0xe30 [mgs] 2014-06-05 13:39:27 [<ffffffffa0d94e9b>] mgs_handle+0x8eb/0x11c0 [mgs] 2014-06-05 13:39:27 [<ffffffffa05db4a1>] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-06-05 13:39:27 [<ffffffffa08db0f8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-06-05 13:39:27 [<ffffffffa05dcfd5>] ? lc_watchdog_touch+0xd5/0x170 [libcfs] 2014-06-05 13:39:27 [<ffffffffa08d2459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-06-05 13:39:27 [<ffffffffa08d31b8>] ? ptlrpc_server_post_idle_rqbds+0x38/0xe0 [ptlrpc] 2014-06-05 13:39:27 [<ffffffffa08dc48e>] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-06-05 13:39:27 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:39:27 [<ffffffff8100c24a>] child_rip+0xa/0x20 2014-06-05 13:39:27 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:39:27 [<ffffffffa08db9c0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-06-05 13:39:27 [<ffffffff8100c240>] ? child_rip+0x0/0x20 2014-06-05 13:39:27 2014-06-05 13:39:27 LustreError: dumping log to /tmp/lustre-log.1402000767.13523 2014-06-05 13:40:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:40:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:40:08 Lustre: 12376:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-145), not sending early reply 2014-06-05 13:40:08 req@ffff880ed811a050 x1470096778199748/t0(0) o400->cc2b1e27-97db-de00-2e98-30a0cc582b24@172.19.1.221@o2ib100:0/0 lens 224/0 e 0 to 0 dl 1402000818 ref 2 fl New:/0/ffffffff rc 0/-1 2014-06-05 13:40:08 Lustre: 12376:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 67195 previous similar messages 2014-06-05 13:40:26 LustreError: 11-0: lse-OST001f-osc-MDT0000: Communicating with 172.19.1.198@o2ib100, operation ost_connect failed with -19. 2014-06-05 13:40:26 LustreError: Skipped 103 previous similar messages 2014-06-05 13:42:14 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.112.223@o2ib5 ns: MGS lock: ffff880ee1982b40/0x49bd4c25c3bfcce0 lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 22 type: PLN flags: 0x20 nid: 192.168.112.223@o2ib5 remote: 0x799417a2fa4663cd expref: 23 pid: 13524 timeout: 4303928908 lvb_type: 0 2014-06-05 13:42:14 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 3671 previous similar messages 2014-06-05 13:45:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:45:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:50:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:50:03 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:50:51 LustreError: 11-0: lse-OST0017-osc-MDT0000: Communicating with 172.19.1.189@o2ib100, operation ost_connect failed with -19. 2014-06-05 13:50:51 LustreError: Skipped 81 previous similar messages 2014-06-05 13:54:12 Lustre: Failing over lse-MDT0000 2014-06-05 13:55:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:55:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 13:55:49 LustreError: 16040:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880f7dece800 x1470095154387196/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 13:55:49 LustreError: 16041:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881fa9ec1400 x1470095154387200/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 13:55:49 LustreError: 16040:0:(qsd_reint.c:58:qsd_reint_completion()) lse-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-5 2014-06-05 13:55:49 LustreError: 16040:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message 2014-06-05 13:57:49 LustreError: 16043:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881fbb2d0c00 x1470095154388344/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 13:57:49 LustreError: 16043:0:(qsd_reint.c:58:qsd_reint_completion()) lse-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x1010000:0x0], rc:-5 2014-06-05 13:57:49 LustreError: 16043:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message 2014-06-05 13:59:38 Lustre: 12307:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3108s); client may timeout. req@ffff880f64681050 x1470096769810936/t0(0) o253->ec5fa9d3-0d7f-af2a-9d07-8dcc3f6052a6@172.19.1.175@o2ib100:0/0 lens 4768/4736 e 0 to 0 dl 1401998870 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 13:59:38 LNet: 10967:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.175@o2ib100 2014-06-05 13:59:38 LNet: 10967:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) Skipped 22 previous similar messages 2014-06-05 13:59:38 LNet: Service thread pid 13512 completed after 3863.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 13:59:38 LustreError: 13512:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-172.19.1.171@o2ib100: deadline 651:1760s ago 2014-06-05 13:59:38 req@ffff880f1eba2050 x1470096768762488/t0(0) o253->d7d712e4-5bf2-0588-bdba-e78cb1893fbf@172.19.1.171@o2ib100:0/0 lens 4768/0 e 1 to 0 dl 1402000218 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:59:38 LustreError: 13512:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 29645 previous similar messages 2014-06-05 13:59:38 Lustre: 12307:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 33095 previous similar messages 2014-06-05 13:59:38 LNet: Service thread pid 12307 completed after 3862.55s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 13:59:38 Lustre: 13512:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:1477s); client may timeout. req@ffff880efd40a850 x1467849453955596/t0(0) o400->10729ad4-5fdf-492b-2f40-a7dcf9f5ea3f@192.168.112.71@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1402000501 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:59:38 LustreError: 12307:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.118.92@o2ib5: deadline 755:1476s ago 2014-06-05 13:59:38 req@ffff880efd4c4050 x1467851089526352/t0(0) o400->6ef6943f-70e4-5e11-7830-5e3cde573894@192.168.118.92@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1402000502 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:59:39 LustreError: 12307:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 27211 previous similar messages 2014-06-05 13:59:39 Lustre: 13512:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 26096 previous similar messages 2014-06-05 13:59:39 Lustre: 13512:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:1712s); client may timeout. req@ffff880ecab2b850 x1467850794314192/t0(0) o250->6ebab674-9d8b-1e69-a9a7-25909b8024b0@192.168.116.108@o2ib5:0/0 lens 400/0 e 0 to 0 dl 1402000267 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:59:39 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.113.211@o2ib5: deadline 755:1057s ago 2014-06-05 13:59:39 req@ffff880ecab68050 x1467849724670004/t0(0) o400->8368bb81-d41e-eb73-a043-158f6fc50bb1@192.168.113.211@o2ib5:0/0 lens 224/0 e 0 to 0 dl 1402000922 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 13:59:40 LustreError: 12376:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 56196 previous similar messages 2014-06-05 13:59:40 Lustre: 13512:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 55527 previous similar messages 2014-06-05 13:59:41 Lustre: MGS: Client 025b7495-aeb0-7362-dd51-9406e444fc82 (at 192.168.117.17@o2ib5) reconnecting 2014-06-05 13:59:41 Lustre: MGS: Client fcc2b94c-1611-2aab-a486-02aa236b124a (at 172.19.1.198@o2ib100) refused reconnection, still busy with 1 active RPCs 2014-06-05 13:59:41 Lustre: Skipped 32 previous similar messages 2014-06-05 13:59:41 Lustre: Skipped 3957 previous similar messages 2014-06-05 13:59:42 Lustre: MGC172.19.1.165@o2ib100: Connection restored to MGS (at 0@lo) 2014-06-05 13:59:42 Lustre: MGS: Client 021a500b-1edf-4ad5-de9e-c1726b5a908c (at 192.168.112.106@o2ib5) reconnecting 2014-06-05 13:59:42 Lustre: Skipped 3490 previous similar messages 2014-06-05 13:59:42 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 13:59:42 Lustre: Skipped 1 previous similar message 2014-06-05 13:59:49 LustreError: 16045:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880ce6f7ac00 x1470095154399212/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 13:59:49 LustreError: 16045:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 1 previous similar message 2014-06-05 13:59:49 LustreError: 16045:0:(qsd_reint.c:58:qsd_reint_completion()) lse-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-5 2014-06-05 13:59:49 LustreError: 16045:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message <ConMan> Console [porter-mds1] log at 2014-06-05 14:00:00 PDT. 2014-06-05 14:00:01 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 14:00:01 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 14:00:07 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:00:07 Lustre: Skipped 1 previous similar message 2014-06-05 14:00:32 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:00:51 LustreError: 11-0: lse-OST000b-osc-MDT0000: Communicating with 172.19.1.178@o2ib100, operation ost_connect failed with -19. 2014-06-05 14:00:51 LustreError: Skipped 100 previous similar messages 2014-06-05 14:00:57 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:00:57 Lustre: Skipped 1 previous similar message 2014-06-05 14:01:05 Lustre: MGS: haven't heard from client 3a3c7f84-2db8-84f8-8ecf-958f82566f52 (at 172.19.1.188@o2ib100) in 2498 seconds. I think it's dead, and I am evicting it. exp ffff882002db9800, cur 1402002065 expire 1402001915 last 1401999567 2014-06-05 14:01:05 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:01:05 Lustre: MGS: haven't heard from client c6c6c633-a600-fc1b-573f-d30a52e6700e (at 172.19.1.202@o2ib100) in 2498 seconds. I think it's dead, and I am evicting it. exp ffff882002557800, cur 1402002065 expire 1402001915 last 1401999567 2014-06-05 14:01:05 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:01:22 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:01:22 Lustre: Skipped 1 previous similar message 2014-06-05 14:01:47 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:01:47 Lustre: Skipped 1 previous similar message 2014-06-05 14:01:49 LustreError: 16162:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880d69254c00 x1470095154400392/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 14:01:49 LustreError: 16162:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 1 previous similar message 2014-06-05 14:01:49 LustreError: 16162:0:(qsd_reint.c:58:qsd_reint_completion()) lse-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-5 2014-06-05 14:01:49 LustreError: 16162:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message 2014-06-05 14:02:12 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:02:12 Lustre: Skipped 1 previous similar message 2014-06-05 14:02:37 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:02:37 Lustre: Skipped 1 previous similar message 2014-06-05 14:03:27 Lustre: lse-MDT0000: Not available for connect from 172.19.1.178@o2ib100 (stopping) 2014-06-05 14:03:27 Lustre: Skipped 3 previous similar messages 2014-06-05 14:03:49 LustreError: 16164:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880fdab2f800 x1470095154401772/t0(0) o101->lse-MDT0000-lwp-MDT0000@0@lo:12/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 2014-06-05 14:03:49 LustreError: 16164:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 1 previous similar message 2014-06-05 14:03:49 LustreError: 16164:0:(qsd_reint.c:58:qsd_reint_completion()) lse-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-5 2014-06-05 14:03:49 LustreError: 16164:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message 2014-06-05 14:04:31 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (651:2053s); client may timeout. req@ffff880f1eba2850 x1470096776102520/t0(0) o253->bea6686c-8f05-839a-09e9-87eef3aabd19@172.19.1.179@o2ib100:0/0 lens 4768/4736 e 1 to 0 dl 1402000218 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 14:04:31 LNet: 10970:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.179@o2ib100 2014-06-05 14:04:31 LNet: 10970:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) Skipped 1 previous similar message 2014-06-05 14:04:31 Lustre: 13523:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 104850 previous similar messages 2014-06-05 14:04:31 LNet: Service thread pid 13523 completed after 2703.56s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 14:04:31 LustreError: 12307:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.120.114@o2ib7: deadline 100:188s ago 2014-06-05 14:04:31 req@ffff880f547a4850 x1461785851008492/t0(0) o250->ba2f0263-3496-feb1-3170-f2c9edee599b@192.168.120.114@o2ib7:0/0 lens 400/0 e 0 to 0 dl 1402002083 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 14:04:31 LustreError: 12307:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 107571 previous similar messages 2014-06-05 14:04:33 LNet: Service thread pid 13528 completed after 4160.57s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 14:04:33 LNet: 10964:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.228@o2ib100 2014-06-05 14:04:33 Lustre: lse-MDT0000: Not available for connect from 192.168.113.118@o2ib5 (stopping) 2014-06-05 14:04:33 Lustre: Skipped 4 previous similar messages 2014-06-05 14:04:34 Lustre: MGS: Client ba2f0263-3496-feb1-3170-f2c9edee599b (at 192.168.120.114@o2ib7) reconnecting 2014-06-05 14:04:34 Lustre: MGS: Client e02ef6c6-7d35-8ebd-59ad-05c4097d6d24 (at 172.19.1.207@o2ib100) refused reconnection, still busy with 1 active RPCs 2014-06-05 14:04:34 Lustre: Skipped 10 previous similar messages 2014-06-05 14:04:34 Lustre: Skipped 469 previous similar messages 2014-06-05 14:04:34 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:04:34 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:04:34 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:04:34 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.112.230@o2ib5 (no target) 2014-06-05 14:04:34 LustreError: Skipped 968 previous similar messages 2014-06-05 14:04:35 Lustre: 13525:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:3408s); client may timeout. req@ffff880f64579050 x1470096781345300/t0(0) o253->e02ef6c6-7d35-8ebd-59ad-05c4097d6d24@172.19.1.207@o2ib100:0/0 lens 4768/4736 e 0 to 0 dl 1401998867 ref 1 fl Complete:/0/0 rc 0/0 2014-06-05 14:04:35 Lustre: 13525:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 2150 previous similar messages 2014-06-05 14:04:35 LNet: 10965:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.207@o2ib100 2014-06-05 14:04:35 LNet: 10965:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) Skipped 2 previous similar messages 2014-06-05 14:04:35 LNet: Service thread pid 13525 completed after 4162.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 2014-06-05 14:04:35 LNet: Skipped 2 previous similar messages 2014-06-05 14:04:37 LustreError: 13520:0:(ldlm_lib.c:2734:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880f64704850 x1468024051488960/t0(0) o256->584a7d4b-d7dc-b7e1-c7bf-ce49bd345450@192.168.112.68@o2ib19:0/0 lens 304/240 e 0 to 0 dl 1401998872 ref 1 fl Interpret:/0/0 rc 0/0 2014-06-05 14:04:37 LustreError: 13520:0:(ldlm_lib.c:2734:target_bulk_io()) Skipped 20 previous similar messages 2014-06-05 14:04:38 LNet: 10964:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) PUT_NACK from 172.19.1.215@o2ib100 2014-06-05 14:04:38 LNet: 10964:0:(o2iblnd_cb.c:381:kiblnd_handle_rx()) Skipped 2 previous similar messages 2014-06-05 14:04:42 Lustre: server umount lse-MDT0000 complete 2014-06-05 14:04:43 Lustre: MGS: Client ba2f0263-3496-feb1-3170-f2c9edee599b (at 192.168.120.114@o2ib7) reconnecting 2014-06-05 14:04:43 Lustre: Skipped 359 previous similar messages 2014-06-05 14:04:47 Lustre: MGS: Client 23ee7d00-4084-051e-e710-70570e1a7236 (at 192.168.120.101@o2ib7) reconnecting 2014-06-05 14:04:55 Lustre: MGS: Client cb59a1f1-dcaf-34dd-893f-5d83b24be88b (at 192.168.112.52@o2ib6) reconnecting 2014-06-05 14:04:55 Lustre: Skipped 12 previous similar messages 2014-06-05 14:05:02 mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 14:05:02 mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) 2014-06-05 14:05:08 Lustre: MGS: Client 1fae2579-8440-9b24-00ba-fc52b1a4f32b (at 192.168.115.67@o2ib10) reconnecting 2014-06-05 14:05:08 Lustre: Skipped 190 previous similar messages 2014-06-05 14:06:17 LustreError: 137-5: lse-MDT0000_UUID: not available for connect from 192.168.114.118@o2ib10 (no target) 2014-06-05 14:06:17 LustreError: Skipped 839 previous similar messages 2014-06-05 14:10:52 LustreError: 11-0: lse-OST0010-osc-MDT0000: Communicating with 172.19.1.182@o2ib100, operation ost_connect failed with -16. 2014-06-05 14:10:52 LustreError: Skipped 504 previous similar messages 2014-06-05 14:11:14 Lustre: lse-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 2014-06-05 14:11:19 Lustre: lse-MDT0000: Will be in recovery for at least 2:30, or until 3670 clients reconnect. 2014-06-05 14:11:35 LustreError: 16780:0:(mdt_open.c:1512:mdt_reint_open()) @@@ [0x2000280e8:0x2db:0x0]/plot_052287->[0x2000233ff:0x2607:0x0] cr_flags=01 mode=0200100000 msg_flag=0x4 not found in open replay. req@ffff881df50c0000 x1451002384608516/t0(115984453232) o101->a1bbdbcc-55a6-fa5e-e94b-700b7afc6bb6@172.16.66.58@tcp:0/0 lens 568/3512 e 0 to 0 dl 1402002756 ref 1 fl Interpret:/4/0 rc 0/0 2014-06-05 14:11:35 LustreError: 16780:0:(mdt_open.c:1512:mdt_reint_open()) Skipped 4 previous similar messages 2014-06-05 14:11:37 Lustre: lse-MDT0000: Denying connection for new client bcc82b95-dc54-a36e-5bbc-480492e7f190 (at 172.16.66.30@tcp), waiting for all 3670 known clients (2720 recovered, 141 in progress, and 0 evicted) to recover in 2:11 2014-06-05 14:11:37 Lustre: Skipped 12 previous similar messages 2014-06-05 14:13:17 Lustre: lse-MDT0000: Denying connection for new client bcc82b95-dc54-a36e-5bbc-480492e7f190 (at 172.16.66.30@tcp), waiting for all 3670 known clients (3494 recovered, 159 in progress, and 0 evicted) to recover in 0:31 2014-06-05 14:13:17 Lustre: Skipped 3 previous similar messages 2014-06-05 14:13:51 Lustre: lse-MDT0000: Recovery over after 2:32, of 3670 clients 3653 recovered and 17 were evicted. 2014-06-05 14:15:31 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.117.202@o2ib5 ns: MGS lock: ffff880f74e85000/0x49bd4c25c45ea1fd lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3669 type: PLN flags: 0x20 nid: 192.168.117.202@o2ib5 remote: 0xf52ce1f7efdb8f2b expref: 9 pid: 13510 timeout: 4305925132 lvb_type: 0 2014-06-05 14:15:31 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 20 previous similar messages 2014-06-05 14:15:31 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.117.106@o2ib10 ns: MGS lock: ffff880f74e85240/0x49bd4c25c45ea1f6 lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3669 type: PLN flags: 0x20 nid: 192.168.117.106@o2ib10 remote: 0x969d64d6cf877f35 expref: 9 pid: 13510 timeout: 4305925132 lvb_type: 0 2014-06-05 14:17:10 Lustre: lock timed out (enqueued at 1402002830, 200s ago) 2014-06-05 14:17:10 LustreError: dumping log to /tmp/lustre-log.1402003030.12371 2014-06-05 14:17:35 LustreError: 138-a: MGS: A client on nid 172.16.66.30@tcp was evicted due to a lock completion callback time out: rc -107 2014-06-05 14:17:54 Lustre: lse-MDT0000: haven't heard from client bcc82b95-dc54-a36e-5bbc-480492e7f190 (at 172.16.66.30@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882008b5c800, cur 1402003074 expire 1402002924 last 1402002847 2014-06-05 14:17:54 Lustre: Skipped 27 previous similar messages 2014-06-05 14:17:54 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:14 LustreError: 13512:0:(ldlm_lib.c:2734:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880f00d0c850 x1450902730063088/t0(0) o256->ac554f9b-4bbd-d88c-807d-d9396eaa3e62@192.168.119.1@o2ib5:0/0 lens 304/240 e 0 to 0 dl 1402003523 ref 1 fl Interpret:/0/0 rc 0/0 2014-06-05 14:19:14 LustreError: 13512:0:(ldlm_lib.c:2734:target_bulk_io()) Skipped 6 previous similar messages 2014-06-05 14:19:17 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.112.65@o2ib19 ns: MGS lock: ffff881e3eae9d80/0x49bd4c25c45fa28e lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3656 type: PLN flags: 0x20 nid: 192.168.112.65@o2ib19 remote: 0x951d20056576607e expref: 15 pid: 13524 timeout: 4306151926 lvb_type: 0 2014-06-05 14:19:17 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 5 previous similar messages 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:17 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:18 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.114.170@o2ib5 ns: MGS lock: ffff881d533c4b40/0x49bd4c25c45f66e4 lrc: 3/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 1863 type: PLN flags: 0x20 nid: 192.168.114.170@o2ib5 remote: 0x9a87c9881213c8c0 expref: 15 pid: 13509 timeout: 4306152093 lvb_type: 0 2014-06-05 14:19:18 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2168 previous similar messages 2014-06-05 14:19:18 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:43 LustreError: 13509:0:(ldlm_lib.c:2734:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880ee48f9050 x1463747476816212/t0(0) o256->54feecad-72a8-c07c-d13c-376d7ada95c4@192.168.112.13@o2ib19:0/0 lens 304/240 e 0 to 0 dl 1402003294 ref 1 fl Interpret:/0/0 rc 0/0 2014-06-05 14:19:48 LustreError: 13518:0:(ldlm_lib.c:2734:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880ee48bf850 x1468024065185444/t0(0) o256->44194fe8-0fb2-e4f6-9136-763b5f2eb6bc@192.168.112.65@o2ib19:0/0 lens 304/240 e 0 to 0 dl 1402003294 ref 1 fl Interpret:/0/0 rc 0/0 2014-06-05 14:19:48 LustreError: 13518:0:(ldlm_lib.c:2734:target_bulk_io()) Skipped 1054 previous similar messages 2014-06-05 14:19:48 LustreError: 166-1: MGC172.19.1.165@o2ib100: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail 2014-06-05 14:19:48 LustreError: 12381:0:(service.c:2000:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.112.48@o2ib19: deadline 100:31s ago 2014-06-05 14:19:49 req@ffff880ed8871050 x1463747476855572/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1402003157 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 14:19:49 Lustre: 13506:0:(service.c:2032:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:31s); client may timeout. req@ffff880ed8873850 x1467850808357288/t0(0) o250-><?>@<?>:0/0 lens 400/0 e 0 to 0 dl 1402003157 ref 1 fl Interpret:/0/ffffffff rc 0/-1 2014-06-05 14:19:49 Lustre: 13506:0:(service.c:2032:ptlrpc_server_handle_request()) Skipped 25 previous similar messages 2014-06-05 14:19:49 LustreError: 12381:0:(service.c:2000:ptlrpc_server_handle_request()) Skipped 2151 previous similar messages 2014-06-05 14:19:49 Lustre: MGS: Client 1fae2579-8440-9b24-00ba-fc52b1a4f32b (at 192.168.115.67@o2ib10) reconnecting 2014-06-05 14:19:49 Lustre: Evicted from MGS (at MGC172.19.1.165@o2ib100_0) after server handle changed from 0x49bd4c25c3fe7c44 to 0x49bd4c25c473ab42 2014-06-05 14:19:49 Lustre: MGC172.19.1.165@o2ib100: Connection restored to MGS (at 0@lo) 2014-06-05 14:19:49 Lustre: Skipped 1 previous similar message 2014-06-05 14:19:49 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:19:53 Lustre: MGS: Client 085e34c3-a0a6-7d94-6a7a-850b3c0ffae1 (at 192.168.112.48@o2ib19) reconnecting 2014-06-05 14:19:58 Lustre: MGS: Client ef602246-72c3-28a2-ad6f-54e41e4218fc (at 192.168.116.128@o2ib5) reconnecting 2014-06-05 14:21:00 Lustre: 17029:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 11s req@ffff881f94c83400 x1459136037249204/t0(0) o400->709e95bf-77e3-be96-a532-eafeb588e0f6@192.168.112.233@o2ib6:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 2014-06-05 14:21:00 LustreError: 11-0: lse-OST0001-osc-MDT0000: Communicating with 172.19.1.167@o2ib100, operation ost_connect failed with -16. 2014-06-05 14:21:00 LustreError: Skipped 1402 previous similar messages 2014-06-05 14:21:00 Lustre: 17029:0:(service.c:1889:ptlrpc_server_handle_req_in()) Skipped 108 previous similar messages 2014-06-05 14:29:48 Lustre: lse-MDT0000: haven't heard from client 7accb070-a661-c0fb-eb1a-5dce5500a1c4 (at 172.16.66.30@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e71a67800, cur 1402003788 expire 1402003638 last 1402003561 2014-06-05 14:29:48 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:31:00 LustreError: 11-0: lse-OST0024-osc-MDT0000: Communicating with 172.19.1.205@o2ib100, operation ost_connect failed with -16. 2014-06-05 14:31:00 LustreError: Skipped 846 previous similar messages 2014-06-05 14:41:01 Lustre: lse-MDT0000: haven't heard from client 7cb97737-a31b-4bf9-64e0-88105a7f1be1 (at 172.16.66.30@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882008a88c00, cur 1402004461 expire 1402004311 last 1402004234 2014-06-05 14:41:01 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline 2014-06-05 14:41:25 LustreError: 11-0: lse-OST0000-osc-MDT0000: Communicating with 172.19.1.248@o2ib100, operation ost_connect failed with -19. 2014-06-05 14:41:25 LustreError: Skipped 329 previous similar messages 2014-06-05 14:45:15 Lustre: lse-OST004f-osc-MDT0000: Connection to lse-OST004f (at 172.19.1.248@o2ib100) was lost; in progress operations using this service will wait for recovery to complete 2014-06-05 14:48:18 Lustre: lse-MDT0000: already connected client lse-MDT0000-lwp-OST004f_UUID (at 172.19.1.248@o2ib100) with handle 0x91a3ac44e76b366a. Rejecting client with the same UUID trying to reconnect with handle 0x700c122eaef5812 2014-06-05 14:48:18 format at ldlm_lib.c:1001:target_handle_connect doesn't end in newline 2014-06-05 14:48:18 Lustre: lse-MDT0000: Rejecting reconnect from the known client lse-MDT0000-lwp-OST004f_UUID (at 172.19.1.248@o2ib100) because it is indicating it is a new client 2014-06-05 14:48:30 Lustre: lse-MDT0000: haven't heard from client lse-MDT0000-lwp-OST004f_UUID (at 172.19.1.248@o2ib100) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d9563f400, cur 1402004910 expire 1402004760 last 1402004683 2014-06-05 14:49:17 Lustre: lse-OST0002-osc-MDT0000: Connection to lse-OST0002 (at 172.19.1.168@o2ib100) was lost; in progress operations using this service will wait for recovery to complete 2014-06-05 14:50:01 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 172.16.66.30@tcp ns: MGS lock: ffff881bddae7b40/0x49bd4c25c49c44f2 lrc: 4/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3657 type: PLN flags: 0x20 nid: 172.16.66.30@tcp remote: 0x197442241c9e170a expref: 7 pid: 12375 timeout: 4307995455 lvb_type: 0 2014-06-05 14:50:01 LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1485 previous similar messages 2014-06-05 14:50:01 LustreError: 12371:0:(ldlm_lockd.c:735:ldlm_handle_ast_error()) ### client (nid 172.16.66.30@tcp) returned 0 from blocking AST ns: MGS lock: ffff880d97419480/0x49bd4c25c4826031 lrc: 4/0,0 mode: CR/CR res: [0x65736c:0x2:0x0].0 rrc: 3657 type: PLN flags: 0x20 nid: 172.16.66.30@tcp remote: 0xb75ad4ec3be1e94a expref: 5 pid: 12376 timeout: 4307995455 lvb_type: 0 2014-06-05 14:50:20 LustreError: 12371:0:(ldlm_lockd.c:735:ldlm_handle_ast_error()) ### client (nid 172.16.66.30@tcp) returned 0 from blocking AST ns: MGS lock: ffff881bddae7b40/0x49bd4c25c49c44f2 lrc: 4/0,0 mode: --/CR res: [0x65736c:0x2:0x0].0 rrc: 3657 type: PLN flags: 0x801000000020 nid: 172.16.66.30@tcp remote: 0x197442241c9e170a expref: 5 pid: 12375 timeout: 4307995455 lvb_type: 0 2014-06-05 14:51:30 LustreError: 11-0: lse-OST0000-osc-MDT0000: Communicating with 172.19.1.249@o2ib100, operation ost_connect failed with -16. 2014-06-05 14:51:30 LustreError: Skipped 54 previous similar messages 2014-06-05 14:53:00 Lustre: lse-OST0037-osc-MDT0000: Connection to lse-OST0037 (at 172.19.1.224@o2ib100) was lost; in progress operations using this service will wait for recovery to complete 2014-06-05 14:53:00 Lustre: Skipped 2 previous similar messages 2014-06-05 14:54:07 Lustre: lse-MDT0000: haven't heard from client 531b7501-3755-9914-7368-09452d85f181 (at 172.16.66.30@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820065d1000, cur 1402005247 expire 1402005097 last 1402005020 2014-06-05 14:54:07 format at client.c:2865:ptlrpc_abort_cond_resched doesn't end in newline <ConMan> Console [porter-mds1] log at 2014-06-05 15:00:00 PDT. 2014-06-05 15:01:55 LustreError: 11-0: lse-OST0031-osc-MDT0000: Communicating with 172.19.1.218@o2ib100, operation ost_connect failed with -16. 2014-06-05 15:01:55 LustreError: Skipped 108 previous similar messages